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ABSTRACT 


Early  Synthetic  Prototyping  (ESP)  is  a  process  and  set  of  tools  that  enable  warfighters  to 
inform  technology  development  and  acquisition  decisions  by  assessing  emerging 
technologies  in  a  game  environment.  Collaborators  in  acquisition,  science  and 
technology,  and  industry  can  develop  models  and  scenarios  for  play  and  assessment.  ESP 
allows  an  unbounded  increase  in  potentially  disruptive  ideas  to  be  explored  at  minimal 
cost  by  inviting  warfighters  at  all  levels  to  drive,  define,  and  refine  future  systems. 

We  conducted  a  study  asking: 

1 .  What  feedback  can  be  gathered  from  game  play? 

2.  Would  that  feedback  be  valuable? 

To  this  end,  groups  of  military  officers  were  engaged  in  several  scenarios  to 
explore  an  unmanned  vehicle  concept  called  Robotic  Wingman.  Through  the  game 
sessions,  players  expressed  ideas  about  the  characteristics  of  a  preferred  interface  and 
how  to  best  employ  Wingman. 

Using  a  game  environment  to  explore  design  concepts  early  in  the  acquisition 
process  can  be  applied  to  early  requirement  refinement  and  rudimentary  trade-off 
analysis.  The  encouraging  results  of  this  preliminary  work  demonstrate  a  strong  potential 
to  leverage  game  environments  to  explore  revolutionary  concepts  to  efficiently  and 
effectively  shape  the  future  of  the  Department  of  Defense. 
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EXECUTIVE  SUMMARY 


The  use  of  modeling  and  simulation  in  aequisition  can  effectively  support  the  acquisition 
of  new  military  technologies.  Rather  than  limit  input  on  proposed  system  requirements 
and  capabilities  to  those  of  leaders  in  a  program  office,  or  experienced  science  and 
technology  representatives,  Early  Synthetic  Prototyping  (ESP)  will  bring  in  input  from 
warfighters  to  develop  future  systems.  A  distributed  game  environment  can  be  leveraged 
as  an  effective  medium  to  bring  warfighters  into  the  development  process. 

ESP  is  a  process  and  set  of  tools  that  enables  warfighters  to  inform  technology 
development  and  acquisition  decisions  by  assessing  emerging  technologies  in  a  game 
environment.  Collaborators  in  acquisition,  science  and  technology,  and  industry  can 
develop  models  and  scenarios  for  play  and  assessment.  ESP  allows  an  unbounded 
increase  in  potentially  disruptive  ideas  to  be  explored  at  minimal  cost  by  inviting 
warfighters  at  all  levels  to  drive,  define,  and  refine  future  systems. 

We  conducted  a  study  at  NPS  asking, 

1 .  What  feedback  can  be  gathered  from  game  play? 

2.  Would  that  feedback  be  valuable? 

To  this  end,  groups  of  military  officers  were  engaged  in  several  scenarios  to 
explore  an  unmanned  vehicle  concept  called  Robotic  Wingman.  Through  the  game 
sessions,  players  expressed  ideas  on  the  characteristics  of  a  preferred  interface  and  how  to 
best  employ  Wingman. 

Using  a  game  environment  to  explore  design  concepts  early  in  the  acquisition 
process  can  be  applied  to  early  requirement  refinement  and  rudimentary  trade-off 
analysis.  The  encouraging  results  of  this  preliminary  work  demonstrate  a  strong  potential 
to  leverage  game  environments  and  explore  revolutionary  concepts  to  efficiently  and 
effectively  shape  the  future  of  the  Department  of  Defense 
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I.  INTRODUCTION 


The  decisive  edge  embodied  in  the  United  States  military  is  sustained,  in  part,  by 
the  continual  pursuit  and  dominance  in  technological  innovation  for  military  applications. 
The  secretary  of  defense,  the  Honorable  Chuck  Hagel,  stated,  “A  world  where  our 
military  lacks  a  decisive  edge  would  be  less  stable  [and]  less  secure  for  both  the  United 
States  and  our  Allies”  (Parker,  2014,  para.  12).  The  United  States  military  has 
historically  employed  its  technologically  superior  resources  to  a  decisive  advantage 
against  its  adversaries.  Modern  antagonists  range  from  technological  peers  to  developing 
nations  and  non-state  actors.  More  nimble  states  that  are  less  encumbered  by  bureaucratic 
institutions  have  a  distinct  advantage  in  the  trajectory  at  which  advancement  can  occur 
despite  a  clear  disadvantage  in  available  resources. 

A.  REQUIREMENTS  FOR  TECHNOLOGICAL  SUPERIORITY 

In  a  cautionary  message  supporting  proposed  improvements  to  the  acquisition 
process,  the  Honorable  Frank  Kendall,  under  secretary  of  defense  for  acquisition, 
technology,  and  logistics  (USD[AT&L])  recently  warned  that  “our  technological 
superiority  is  very  much  at  risk”  (Freedberg,  2014,  para.  1).  The  ongoing  effort  to  seek 
innovative  technological  solutions  for  current  and  future  problems  must  include  the 
Department  of  Defense  (DOD),  science  and  technology  (S&T),  and  industry  to  keep 
ahead  of  any  adversary  posing  a  threat,  whether  that  be  a  first-world  peer  or  a  less 
sophisticated  entrant. 

To  maintain  superiority  in  the  face  of  a  nimble  opposition,  the  DOD  must  foster 
the  development  and  procurement  of  technology  to 

•  remain  agile, 

•  explore  a  multitude  of  options  through  prototyping,  i  and 

•  conduct  appropriate  and  meaningful  evaluation  of  options. 

1  As  discussed  in  Chapter  II,  Section  B,  this  thesis  takes  a  broad  view  of  “prototyping”  that  includes 
both  conventional  physical  prototypes  as  well  as  virtual  prototypes  of  varying  fidelity  throughout  the 
acquisition  pipeline. 
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In  meeting  these  requirements,  the  DOD  will  be  able  to  aehieve  and  maintain 
teehnologieal  superiority  through  the  development  and  procurement  of  the  systems 
deemed  necessary  to  meet  warfighter  needs,  now  and  in  the  future.  With  the  assumption 
that  these  three  specific  areas  of  interest  are  necessary  to  support  advancing  technological 
superiority,  the  DOD  must  develop  a  new  means  to  identify,  prototype,  and  assess 
emerging  technologies,  ideas,  and  concepts.  This  thesis  is  based  on  an  assumption  that 
traditional  acquisition,  while  important,  is  inadequate  to  meet  this  emerging  need.  The 
DOD  acquisition  process  needs  something  new  that  expands  the  role  of  our  most 
important  asset:  the  warfighter. 

A  distributed  game  environment  is  proposed  as  a  promising  collaborative  venue 
for  reliable  evaluation  of  future  concepts.  Secretary  Hagel  supported  the  ability  to  “assess 
which  commercial  innovations  have  military  potential  . . .  rapidly  adopt  them  and  adapt 
them,  then  test  and  refine  them,  including  through  war-gaming  and  demonstrations” 
(Lyle,  2014,  para.  25).  This  effort  can  be  accomplished  for  commercial  innovations, 
proposed  military  systems,  and  the  necessary  mission  context  for  those  systems  within 
the  game  environment  proposed  for  Early  Synthetic  Prototyping  (ESP). 

B.  EARLY  SYNTHETIC  PROTOTYPING:  AN  ENVIRONMENT  TO 

FOSTER  INNOVATION 

ESP  is  a  process  and  set  of  tools  proposed  by  the  Army  Capabilities  Integration 
Center  (ARCIC)  Army  Capabilities  as  a  means  to  aid  innovation.  More  importantly,  ESP 
will  bring  in  warfighters  as  a  source  of  creativity  and  assessment.  ESP  proposes  a 
distributed  game  network  where  virtual  (or  synthetic)  prototypes  can  be  quickly  built  and 
tested  in  realistic  scenarios.  Warfighters  can  “play”  the  prototypes  in  the  scenarios  and 
also  introduce  modifications  for  new  prototypes  or  concepts.  The  output  is  notional 
performance  data  of  the  prototypes  for  assessment  and  further  application  to  system 
development.  The  ESP  concept  is  described  in  detail  in  Chapter  3. 

To  attain  the  best  system  solution,  ESP  looks  beyond  the  traditional  program 
acquisition  unit  cost  (PAUC),  which  is  the  program  acquisition  cost  divided  by  the 
quantity  procured  (Defense  Acquisition  University,  2014a).  ESP  aims  to  increase  the 
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value  of  each  unit  rather  than  strictly  reduce  the  numerator,  which  encompasses  the  total 
cost  of  the  development  and  acquisition  effort.  With  ESP,  the  DOD  will  get  the  best 
possible  systems  to  warfighters,  thereby  improving  overall  value  of  the  system. 
Presumably,  if  the  right  system  is  developed  and  procured,  it  has  the  potential  to  save 
lives  or  support  successful  engagements.  Therefore,  the  high  value  system  can  reduce  the 
need  for  additional  or  replacement  systems  to  accomplish  the  same  mission  over  an 
extended  duration. 

In  a  distributed  game  environment,  ESP  offers  the  opportunity  to  develop 
synthetic  prototype  solutions  rapidly  and  push  them  out  for  collaboration  and  evaluation 
within  weeks  or  days,  rather  than  months  or  years.  Instead  of  limiting  the  number  of 
prototypes  or  options  to  be  considered  based  on  cost  or  available  time,  ESP  creates  an 
environment  for  unbounded  development  opportunities.  Major  defense  acquisition 
programs  (MDAPs),  historical  and  recent,  demonstrate  how  the  evaluation  of  full-scale 
physical  prototypes  can  carry  considerable  cost  in  funding  and  time.  Eor  example,  in  the 
video  titled  “Battle  of  the  X-planes,”  the  quantity  of  physical  prototypes  to  be  evaluated 
was  limited  to  two  competitors  (Nova,  2003).  More  recently,  the  Joint  Eight  Tactical 
Vehicle  evaluated  three  competing  prototypes  that  cost  more  than  $177  million  and  27 
months  (East,  2014).  The  Eittoral  Combat  Ship  effort  assessed  prototypes  from  two 
competitors  expending  over  $1  billion  and  72  months  (East,  2014).  In  contrast  to  these 
MDAPs,  future  programs  can  employ  ESP  to  substantially  increase  the  number  of 
prototypes  to  be  evaluated  early  in  the  process  and  in  a  cost-effective  virtual 
environment.  Synthetic  prototypes  can  also  be  virtually  run  through  multiple  scenarios 
for  additional  simulated  evaluation  in  variant  conditions  and  environments  without  the 
cost  of  bent  metal  or  a  live  exercise  or  war  game. 

Within  ESP,  the  community  available  to  contribute  to  a  reliable  evaluation  of 
proposed  technology  solutions  is  nearly  unbounded.  Eurthermore,  this  community  can 
bring  a  previously  untapped  breadth  and  depth  of  experience  to  offer  realistic  and 
experienced  perspective  on  how  a  proposed  system  could  be  best  employed.  Within  the 
DOD,  services  can  work  collaboratively  to  develop  a  bevy  of  proposed  solutions  and 
collect  notional  performance  data  in  fixed  scenarios,  recommendations  for  system 
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modification  and  resulting  performance  ehanges,  and  coneepts  for  system  employment 
and  tactics  beyond  existing  scenarios.  The  evaluators  in  ESP  are  not  limited  to 
professionals  in  S&T  or  aequisition,  but  rather  are  open  to  the  warfighters  with  the 
taetieal  and  operational  experienee  to  provide  valid  input  to  develop  and  refine  proposed 
system  solutions. 

C.  EARLY  SYNTHETIC  PROTOTYPING:  THE  VALUE  PROPOSITION  OF 

THIS  THESIS 

ESP  ean  be  successful  only  if  the  assessment  component  is  trusted  and  reliable.  It 
is  not  enough  that  warfighters  can  “play”  with  virtual  prototypes  early  in  the  design 
proeess  if  researchers  are  unable  to  learn  anything  useful  that  ean  be  put  to  immediate  use 
to  improve  the  aequisition  program  or  eoneept.  Therefore,  this  thesis  is  focused  on  how  to 
assess  prototypes  in  a  distributed  game  network:  What  feedbaek  ean  be  gleaned,  and  is  it 
useful  to  decision-makers? 

Systems  are  developed  and  aequired  within  the  DOD  Decision  Support 
Eramework,  and  ESP  will  become  an  additional  option  within  this  existing  construct. 
Prototyping  has  already  been  incorporated  as  a  useful  tool  and  is  mandated  in  some  cases. 
Simulation  efforts  within  aequisition  are  not  new,  and  ESP  can  gain  valuable  insight  from 
previous  efforts.  To  make  ESP  viable,  a  large  number  of  contributors  is  essential.  Game 
analytic  techniques  should  prove  useful  in  colleeting  and  evaluating  the  data  aceumulated 
from  warfighter  game  play  for  evaluation  and  digestion  by  deeision-makers. 

This  thesis  reports  on  a  study  undertaken  at  the  Naval  Postgraduate  School  to 
explore  whether  a  distributed  game  environment  is  an  appropriate  venue  to  conduct  a 
reliable  evaluation.  Eurthermore,  we  investigated  the  forms  that  assessment  data  might 
take,  yielding  insights  to  the  next  steps  in  determining  how  game  analyties  and  other 
teehniques  might  be  used  to  seamlessly  and  unobtrusively  collect  assessment  data  from 
game  players,  whieh  could  guide  future  aequisitions. 

In  summary,  the  hypothesis  of  ESP  is  that  value,  rapid  development, 
consideration  of  multiple  options,  and  the  eondueting  of  reliable  evaluation  are  neeessary 
to  achieve  and  maintain  teehnologieal  superiority.  This  thesis  explores  one  portion  of  that 
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premise,  arguing  that  reliable  assessments  ean  be  accomplished  through  the  accumulation 
of  warfighter  input  through  game  play  of  proposed  systems  in  a  synthetic  environment. 
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II.  BACKGROUND 


Early  Synthetic  Prototyping  touches  multiple  facets  of  system  development, 
starting  with  the  acquisition  framework  itself.  Within  the  acquisition  umbrella,  ESP  will 
have  multiple  stakeholders:  users,  designers,  modeling  and  simulation  professionals,  and 
science  and  technology  representatives.  ESP  will  use  a  game  environment  to  interact  with 
these  stakeholders,  so  the  ability  to  collect,  process,  and  disseminate  useful  data  is 
essential  to  making  the  warfighter  accessible  game  environment  a  useful  tool  for 
decision-makers. 

A.  ACQUISITION 

Acquisition  programs  in  all  uniformed  services  struggle  to  meet  prescribed 
timelines,  remain  within  budget,  and  retain  the  agility  needed  to  meet  looming,  but 
unknown,  requirement  changes.  The  DOD  budget  request  for  fiscal  year  (EY)  2012 
totaled  over  $553  billion;  over  $203  billion  of  that  total  was  designated  for  procurement 
and  research,  development,  test,  and  evaluation  (RDT&E)  programs  (Office  of  the 
Secretary  of  Defense  [Comptroller],  2011).  hollowing  a  decade  of  persistent  combat 
necessitating  high  expenditures,  the  future  DOD  budget  will  be  significantly  reduced 
(Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Eogistics  [USD(AT&E)], 
2014).  This  reduction  makes  prioritizing  efficiency  an  imperative  for  acquisition.  Of  note, 
the  president’s  EY2015  budget,  seen  in  Eigure  1,  shows  that  procurement  and  RDT&E 
combined  will  fall  to  $153.9  billion,  meaning  that  S&T  experts  must  continue  to  develop 
solutions  with  a  smaller  budget  (USD[AT&E],  2014).  See  Appendix  A  for  additional 
historical  budget  data. 


7 


by  Budget 
Account  MilCon 


Other 

$2.4 


by  Military 
Department 


NOTE;  Budget  amounts  are  In  billions  of  then-year  (unadjusted)  dollars.  OCO  Is  not  IrKluded.  'MilCon'  is  Military 
Construction. 

Figure  1.  2015  DOD  Budget  Breakout  (from  USD[AT&L],  2014) 

Aneedotally,  warfighters  report  that  it  takes  too  long  to  develop  and  aequire 
teehnology  solutions  for  the  military  and,  upon  reeeipt,  system  eapabilities  might  be 
exeeeded  by  those  of  produets  proeured  eommereially  off  the  shelf  for  lower  eost.  The 
Government  Aeeountability  Office  (GAO)  report  on  Defense  Acquisition  (2012)  backs 
up  that  anecdotal  data,  revealing  that  for  96  major  acquisition  programs  in  place  in 
FY2011,  total  acquisition  costs  exceeded  estimates  by  over  $74.4  billion  (GAO,  2012). 
Research  and  development  costs  contributed  $14  billion  to  that  total  overage.  The  GAO 
report  further  noted  that  over  the  course  of  FY2011,  the  average  delay  of  in-progress 
programs  increased  by  one  month;  this  brings  the  overall  average  delay  to  23  months 
when  compared  to  a  program’s  initial  full  estimate  (2012).  Programs  contributing  to  the 
$14  billion  increase  in  research  and  development  costs  were  found  to  be  in  production  or 
using  concurrent  development  and  production  strategies.  Justifications  for  the  cost 
increases  ranged  across  the  following;  “reduce  risk,”  “meet  requirements,” 
“modernization,”  “correction  of  deficiencies,”  “new  capabilities  and  testing,”  and 
“software  development”  (GAO,  2012,  p.  10). 

Given  the  nature  of  the  reported  cost  increases,  the  additional  requirements 
identified  in  FY201 1  may  have  been  anticipated  and  avoided  through  the  conducting  of  a 
more  thorough  requirements  analysis  prior  to  declaration  of  cost  and  time  estimates  based 
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on  acquisition  strategies.  The  ability  to  clearly  identify  requirements  and  all  associated 
costs  is  critical  to  a  successful  acquisition  strategy,  but  defense  programs  are  challenged 
by  the  need  to  predict  the  unpredictable  at  an  early  stage  in  the  acquisition  cycle. 

1,  Department  of  Defense  Decision  Support  System 

The  DOD  Decision  Support  System,  represented  in  Figure  2,  also  referred  to  as 
the  Big  “A”  concept,  can  be  a  complicated  process  to  navigate.  The  process  map  for  the 
Decision  Support  System  is  notoriously  complex  and  derisively  referred  to  the  “horse 
blanket”  (Defense  Acquisition  Portal,  2010).  The  level  of  detail  required  to  successfully 
take  a  program  of  record  from  the  “great  idea”  stage  through  fielding  and  employment 
and  then  to  eventual  retirement  requires  a  knowledgeable  and  experienced  team  of 
acquisition  professionals.  The  proposal  for  ESP  in  no  way  seeks  to  add  a  mandatory 
event  for  program  managers  within  the  already  lengthy  list  of  to-do  requirements  that 
must  be  fulfilled  for  any  MDAP.  Rather,  ESP  is  a  supplementary  tool  to  be  used  within 
the  acquisition  framework  to  support  the  effort  of  a  service  or  program  to  refine  and 
develop  the  best  possible  system  to  support  the  warfighter. 


^ Joint  Capabilities  ~ 
Integration  and 
Developmefit  System 
(JCIOS) _ 


Planning, 

Programming. 

Budgeting, 
and  Execution 
(PPBE)  > 

Process 


Defense 

Acquisition 

System 


Eigure  2.  Abstraction  of  the  DOD  Decision  Support  System,  also  Known  as 
the  Big  “A”  (from  Defense  Acquisition  Elniversity  [DAU],  2014a) 


The  Big  “A”  framework  shown  in  Eigure  2  is  not  limited  to  the  Defense  Acquisition 
System  (DAS).  The  DAS  is  one  of  three  interdependent  decision  support  systems,  with  the 
others  being  the  Joint  Capabilities  Integration  and  Development  System  (JCIDS)  and  the 
Planning,  Programming,  Budgeting,  and  Execution  (PPBE)  process.  JCIDS  supports  the 
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Joint  Requirements  Oversight  Couneil  (JROC)  and  the  Chairman  of  the  Joint  Chiefs  of  Staff 
(CJCS)  and  focuses  on  identifying  warfighting  capability  gaps  that  could  be  filled  by  either  a 
materiel  solution  or  non-materiel  change  to  the  doctrine,  organization,  training,  leadership, 
personnel,  facilities,  or  policy  (DOTMLPF-P).  JCIDS,  therefore,  informs  the  decision  on 
solutions,  materiel  or  otherwise,  needed  in  support  of  a  particular  capability  and  seeks  to 
effectively  identify,  assess,  validate,  and  prioritize  joint  capability  requirements.  Where  non¬ 
materiel  solutions  are  insufficient,  the  DAS  is  called  into  play  to  identify  and  procure  an 
effective  materiel  solution  to  offset  the  gap  in  capability.  Figure  3  shows  the  interaction 
between  the  JCIDS  and  the  DAS  and  the  points  at  which  JCIDS  documents  are  incorporated 
in  the  DAS  process.  The  PPBE  system  completes  the  triad  by  resourcing  the  requirements 
determined  in  the  JCIDS  and  DAS  in  conjunction  with  the  mandates  of  the  president’s 
National  Security  Strategy.  PPBE  is  the  process  by  which  activities  output  from  the  JCIDS 
and  DAS  are  funded  for  development,  fielding,  and  sustainment,  as  well  as  prioritization 
against  other  requirements  (Office  of  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  [OUSD(AT&L)],  2013). 


10 


2,  Joint  Capabilities  Integration  Development  System 

JCIDS  documents  are  integrated  into  the  phases  of  aequisition  as  a  linkage  from 
the  eapability  requirement,  as  seen  in  Figure  3.  The  five  phases  of  the  DAS  are  Materiel 
Solution  Analysis  (MSA),  Technology  Development  (TD),  Engineering  and 
Manufaeturing  Development  (EMD),  Produetion  and  Deployment  (P&D),  and 
Operations  and  Support  (O&S).  The  doeuments  assoeiated  with  the  JCIDS  process  are 
depieted  in  Eigure  4  as  pink  reetangles  amid  the  green  squares  marking  sponsor  activities 
and  blue  triangles  indieating  aequisition  deeisions.  The  documents  inelude  the  Initial 
Capabilities  Document  (ICD),  the  Capability  Development  Documents  (CDDs),  and 
Capability  Production  Documents  (CPDs;  CJCS,  2012). 


3,  Defense  Acquisition  System 

The  DAS,  as  seen  in  Eigure  5,  is  a  proeess  guided  by  events.  The  materiel 
development  decision  (MDD)  is  the  initial  entry  point  into  the  DAS.  There  are  three 
milestone  reviews — ^A,  B,  and  C — and  several  additional  decision  points  within  each 
phase.  Milestone  Decision  A  is  characterized  by  a  eonelusive  materiel  solution  analysis 
that  indicates  a  direction  to  be  taken  toward  effeetive  development  of  a  suitable  solution. 
Milestone  Deeision  B  is  characterized  by  completion  of  the  development  of  appropriate 
technology  as  well  as  the  eonduet  of  a  preliminary  design  review.  Milestone  Deeision  C 
oeeurs  at  the  elose  of  the  EMD  phase  and  is  characterized  by  a  program  prepared  to  send 
a  system  into  low  rate  initial  produetion  (TRIP)  with  plans  for  introduetory  operations, 
training,  and  edueation  leading,  ereating  an  initial  operational  eapability  (IOC)  for  the 
warfighter.  Subsequently,  a  full  rate  production  (ERP)  decision  is  made  and  the  status  of 
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full  operational  capability  (FOC)  is  achieved  as  the  system  transitions  into  the  post¬ 
acquisition  sustainment  phase  (DAU,  2014a). 
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Figure  5.  The  Defense  Acquisition  System  (from  DAU,  2014a) 


4,  Planning,  Programming,  Budgeting,  and  Execution 

Without  sufficient  resources  to  enable  the  process,  acquisition  is  incomplete.  The 
PPBE  process  takes  its  guidance  from  the  president  and  his  national  security  strategy,  and 
funding  is  authorized  and  appropriated  by  Congress.  Since  2003,  the  budget  has  been  put 
together  for  two-year  periods  to  effectively  correspond  to  the  president’s  four-year  term, 
as  detailed  in  Figure  6  (DAU,  2014a).  In  the  Quadrennial  Defense  Review  (QDR),  the 
president  sets  the  agenda  for  defense  spending  over  the  duration  of  his  term.  The 
programs  budgeted  for  by  each  service  correspond  to  the  QDR  priorities.  The  Program 
Objective  Memorandum  (POM)  process  provides  the  services  an  opportunity  to  prioritize 
resource  requirements  and  compete  for  available  resources  to  fund  those  priorities. 
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PPBE  Two-Year  Cycles  Corresponding  to 
Four- Year  Presidential  Terms 


1  (Ri/.if  .  .>nd  Refiutif)  -  :ir; 

New  National  Security  Strategy 

Off-year  JPG  as  required  (at  discretion  of  SECDEP) 

Limited  Changes  to  Baseline  Program 

I !  I ,  lir-- Vi.-  ■w;..nd;'l. 

Quadrennial  Oetense  Review  (OOR) 

-  Aligned  with  PB  submission  in  second  year  of  an  administration 
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Figure  6.  Planning,  Programing,  Budgeting,  and  Exeeution 

(from  DAU,  2014a) 


Defense  spending  has  been  reeeding  after  a  robust  deeade  of  spending  in  support 
of  ongoing  eombat  operations.  As  a  result  of  spending  reduetions,  effeetive  prioritization 
is  eritieal  to  eaeh  serviee  to  ensure  that  eaeh  is  able  to  meet  eapability  requirements  in  the 
most  effective  and  efficient  way  possible. 

In  addition  to  the  overarching  Big  “A”  concept  and  processes,  there  are  best 
practices  that  evolve  to  support  procurement  of  the  best  possible  resources.  Prototyping  is 
one  of  those  best  practices.  The  Weapons  Systems  Acquisition  Reform  Act  (WSARA)  of 
2009  formalized  prototypes  into  the  acquisition  cycle  through  a  mandate  for  competitive 
prototyping. 

B.  PROTOTYPING 

Prototyping  has  become  an  integral  element  in  the  acquisition  process,  though 
there  is  not  a  strict  framework  for  how  it  must  be  accomplished  for  individual  programs. 
Competitive  prototyping  is  the  only  version  with  statutory  requirements  for  MDAPs. 
There  are  additional  options  beyond  competitive  prototyping  that  can  support  acquisition 
efforts  through  the  process. 
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In  the  2012  GAO  report,  the  eomptroller  general  of  the  United  States,  Gene 
Dodaro,  pointed  out  that  on  a  positive  note,  several  MDAPs  are  employing  strategies 
sueh  as  competitive  prototyping  in  an  effort  to  control  costs.  Competitive  prototyping  is 
required  by  the  WSARA  of  2009.  Using  prototyping  to  “reduce  technical  risk,  refine 
requirements,  validate  designs  and  cost  estimates,  and  evaluate  manufacturing  processes” 
(GAO,  2012,  p.  30)  can  ideally  mitigate  cost  increases  later  in  the  acquisition  process. 
The  Defense  Acquisition  Guidebook  (DAG)  highlights  the  benefits  of  prototyping  by 
pointing  out  that  prototypes  could  potentially  support  acquisition  of  “more  innovative 
solutions  at  better  value”  (DAU,  2014a,  para.  4.2.4).  The  DAG  goes  on  to  lay  out  a 
flexible  guideline  that  the  competitive  prototype  can  focus  either  on  a  full  system  or 
portions  thereof,  such  as  critical  technology  or  system  elements. 

1,  Competitive  Prototyping 

Some  see  early  prototyping  as  a  potential  pitfall  and  limiting  factor  rather  than  as 
an  enabler.  A  prototyping  skeptic  would  argue  that  early  exposure  limits  the  aperture  of 
possibilities  and  stifles  creativity.  From  a  design  perspective,  this  argument  may  have 
merit;  however,  competitive  prototyping  aims  to  reveal  multiple  potential  solutions,  each 
a  unique  approach  to  meeting  the  system  requirements.  Competitive  prototyping  is  a 
mandate  in  the  WSARA  of  2009  during  the  technology  development  phase  prior  to 
Milestone  B  decisions  where  the  Milestone  Decision  Authority  grants  entry  into  the 
Engineering  and  Manufacturing  Development  phase  (WSARA,  2009).  Competitive 
prototyping  invites  competition  from  commercial  vendors  but  does  not  explicitly  attempt 
to  bring  in  novel  or  leap-ahead  solutions.  Competitive  prototyping  can  result  in  great 
motivation  for  vendors  to  provide  innovative  solutions,  and  can  help  ensure  acquisition 
dollars  are  spent  toward  development  and  evaluation  of  a  near-final  production  solution. 
However,  costs  can  quickly  escalate  if  requirements  creep  occurs  during  this  phase.  Costs 
associated  with  competitive  prototyping  include  producing  a  mockup  of  the  system  or 
sub-systems  for  evaluation  against  competitors  (WSARA,  2009).  The  WSARA  affords  a 
program  manager  the  opportunity  to  request  a  waiver  if  the  expense  of  a  competitive 
prototype  is  not  economical,  given  the  anticipated  system  life-cycle  cost.  ESP  can  bring 
the  positive  features  of  competitive  prototyping,  such  as  requirements  refinement,  into 
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the  low  cost,  government-directed  tradespace  of  a  virtual  environment  before  a  physical 
prototype  is  produced.  The  ability  to  refine  requirements  facilitates  pre -Milestone  A 
concept  refinement  and  supports  acquisition  process  improvement. 

William  Fast  (2014)  made  a  concerted  effort  to  evaluate  fairly  the  effectiveness, 
or  lack  thereof,  of  competitive  prototyping.  Fast  looked  at  programs  from  1990  through 
2012  that  “demonstrated  technology  maturity  on  prototypes  in  a  relevant  environment” 
(p.  469);  indicated  a  Technology  Readiness  Level  of  6  (TRL6);  and,  also  accomplished  a 
preliminary  design  review  before  Milestone  B.  His  research  revealed  that  programs  that 
met  these  two  criteria  “more  often  to  show  negative  PAUC  cost  growth”  (p.  469). 

Fast  (2014)  was  also  able  to  show  that  these  programs  were  not  less  likely  to 
suffer  a  schedule  breach.  The  seemingly  mixed  results  of  this  effort  are  further  muddled 
by  process  change  that  was  incorporated  during  the  researched  timeframe  as  well  as 
variant  criteria  used  in  previous  research.  The  results  echo  the  findings  obtained  by 
Jeffrey  Drezner  in  a  1992  RAND  study.  Drezner  concluded  that  the  “effect  of  prototyping 
on  program  cost,  schedule,  and  performance  is  ambiguous  due  to  the  effect  of 
confounding  variables”  (p.  68).  Ultimately,  in  the  absence  of  clear  quantification  of 
competitive  prototyping  benefits,  the  acquisition  can  look  back  to  Drezner’ s 
recommendation  that  each  program  manager  must  “weigh  the  cost  and  benefits  of 
prototyping  against  the  risks  and  consequences  of  proceeding  into  the  next  phase  without 
the  knowledge  gained  from  prototypes”  (Fast,  2014,  p.  467).  It  is  easy  to  understand  why 
many  acquisition  professionals  continue  to  consider  the  risk  mitigation  achieved  through 
seeking  out  and  evaluating  prototypes  more  valuable  than  the  possible  limitations  of 
employing  a  prototype. 

2,  Other  Prototyping  in  Acquisition 

Competitive  prototyping  is  not  the  only  type  of  prototype  that  can  be  employed  to 
support  successful  system  development.  Defense  Systems  Management  College  (DSMC) 
Military  Fellows  Garcia,  Gocke,  and  Johnson  proposed  virtual  prototyping  as  far  back  as 
1994.  Garcia  et  al.  (1994)  produced  a  comprehensive  product  that  factored  in  the  post- 
Cold  War-reduced  military  funding  environment  along  with  the  spectrum  of  technology 

available  to  support  creation  and  evaluation  of  virtual  prototypes  within  synthetic 
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environments  to  complement  ongoing  acquisition  efforts.  Binder  (1998)  proposed  an 
implementation  of  virtual  prototyping  for  industry  in  1998  to  support  efforts  of  engineers 
to  build  a  model  and  simulate  its  moving  parts  to  optimize  designs  before  the  more  costly 
action  of  building  a  physical  prototype  or  proceeding  directly  to  manufacturing.  Virtual 
prototypes  can  be  used  to  model  and  review  systems  before  making  a  commitment  to 
evaluate  a  physical  prototype  or,  alternately,  eliminate  the  need  for  the  physical  prototype 
altogether  given  substantial  risk  mitigation  to  offset  the  inherent  limitations  of  virtual 
renditions  of  systems. 

Given  the  structured  environment  employed  to  ensure  that  the  nation’s  needs  are 
met  within  available  resources  in  a  fiscally  responsible  manner,  prototyping  can  be  an 
integral  part  in  the  acquisition  of  new  resources  for  the  DOD.  Hencke  (2014)  proposed 
that  prototyping  is  maturing  from  a  design  tool  into  “a  collection  of  developmental  and 
experimental  activities  that  are  maximizing  the  value  of  developing  and  working  with 
intermediate  forms  (models  or  demonstrators)”  (p.  11).  He  divided  prototypes  into  two 
distinct  instruments:  developmental  and  operational.  Figure  7  shows  functions  of  each 
instrument  and  alignment  within  acquisition  and  technology  readiness.  Hencke  (2014) 
pointed  out  that  there  is  an  additional  requirement  for  prototyping  “to  the  left  . . .  where 
problem  definition  and  concept  development  reside”  (p.  14). 
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barriers.  •  Demonstrate  robust  fabrication  processes. 
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Figure  7.  Prototype  Instruments  and  Methodology  (after  Hencke,  2014) 
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Expressing  support  for  prototypes  in  support  of  innovation,  Alan  Shaffer,  an  aid 
to  the  under  seeretary  of  defense  for  Aequisition,  Technology,  and  Logistics 
(USD[AT&L]),  argued  that  prototypes  are  essential  in  the  early  stages  “before  [going]  to 
formal  lockdown  of  requirements  at  Milestone  B”  or  Critical  Design  Review  because, 
after  that  point,  it  is  challenging  to  incorporate  additional  S&T  because  a  “program  must 
focus  on  schedule,  budget,  and  getting  new  technologies  to  work  reliably”  (Freedberg, 
2014,  para.  9). 

C.  SIMULATION  IN  ACQUISITION 

There  have  been  several  attempts  to  incorporate  simulation  or  virtual  prototypes 
into  the  acquisition  process  to  include  efforts  to  leverage  simulation  to  reduce  costs  in 
system  design  as  well  as  in  test  and  evaluation. 

1.  Simulation-Based  Acquisition 

Simulation-Based  Acquisition  (SBA)  debuted  in  1997  as  a  concept  to  incorporate 
powerful  aspects  of  simulations  in  support  of  acquisition.  ESP  appears  to  be  very  similar 
to  SBA;  however,  there  are  two  important  distinctions:  (a)  ESP  is  focused  on  early 
concept  development,  when  costs  are  relatively  low  but  when  it  is  critical  to  get  major 
design  decisions  right;  and  (b)  ESP  allows  for  the  consideration  of  more  design 
alternatives  than  SBA  or  any  known  acquisition  process  options  by  many  orders  of 
magnitude.  ESP  operates  on  the  premise  that  disruptive  ideas  are  more  likely  to  appear 
when  10,000  design  variations  are  considered  rather  than  just  10. 

SBA  promised  a  reduction  in  time,  resources,  and  risk  associated  with  the 

acquisition  process  and  an  increase  in  the  quality,  military  utility,  and  supportability  of 

the  systems  it  fielded  (National  Research  Council,  2002;  Sanders,  1997).  Despite  its  merit 

as  a  means  to  reduce  expenditures  by  introducing  modeling  and  simulation  into  the 

acquisition  cycle,  SBA  achieved  limited  success  partly  because  it  attempted  to  address 

the  entire  life  cycle  from  idea  inception  through  production,  fielding,  and  employment. 

The  use  of  simulation  at  each  phase  of  the  acquisition  process  differs.  Rather  than  take  a 

holistic  approach  to  simulation  in  acquisition,  ESP  seeks  to  address  the  unique  needs  of 

the  early  phases.  The  ESP  approach  also  serves  to  shift  the  bulk  of  the  change  demand, 
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particularly  the  demand  eoming  from  the  warfighters  ultimately  expeeted  to  employ  the 
system,  from  post-Milestone  C,  when  it  is  expensive  to  make  ehanges,  to  pre-Milestone 
A,  when  it  is  far  eheaper. 

2,  Simulation  and  Modeling  for  Acquisition,  Requirements,  and 
Training 

Simulation  and  Modeling  for  Aequisition,  Requirements,  and  Training  (SMART) 
attempted  to  model  physieal  properties  and  assoeiated  eosts  in  a  virtual  environment  at  an 
early  stage  and  earry  those  through  the  fielding  and  training  phases  (Davis,  1999).  The 
holistie  approaeh  of  SMART  offers  a  eontrast  to  the  focus  of  ESP.  By  limiting  its  focus 
to  the  early  design  phase,  ESP  has  greater  latitude  regarding  the  level  of  preeision 
required  to  explore  an  early  coneept  as  eompared  to  detailed  design  deeisions  later  in  the 
aequisition  proeess.  During  concept  exploration,  the  fidelity  of  a  ballistie  model,  for 
example,  is  of  less  eoneern  than  determining  whether  a  weapon  system  used  in  a  speeifie 
way  merits  further  study. 

3,  Dragonfly 

The  DAU  is  using  a  produet  ealled  Dragonfly  as  a  teaehing  aid  in  the  program 
manager  eourse  (DAU,  2010).  Dragonfly  simulates  the  tradespaee  with  realistie  portrayal 
of  eost  and  performanee  faetors.  The  interfaee  to  the  environment,  as  seen  in  Eigure  8,  is 
easy  to  use,  allowing  a  player  to  seleet  preferred  eomponents  and  then  weigh  assoeiated 
eapability  against  eost.  System  performanee  is  evaluated  head-to-head  against  peers  in 
the  Dragonfly  virtual  environment.  The  diversity  in  seleeted  teehnical  solutions  presents 
an  opportunity  to  explore  a  variety  of  solutions  in  a  standard  trial  environment  (see  SEA 
in  the  next  seetion.  Chapter  II,  Section  C,  Part  4,  which  proposed  this  as  well).  The 
potential  for  multiple  optima  emulates  the  erowdsoureing  objeetive  of  ESP  and  provides 
a  space  to  refine  requirements  against  possible  and  plausible  teehnieal  solutions. 
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DESIGN  AAODE:  COAAPONENT  SELECTION 
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Figure  8.  Dragonfly  User  Interface  (from  DAU,  2010,  p.  8) 


A  doctoral  dissertation  by  Major  Josh  Keena  (2011)  used  MindRover,  the 
precursor  to  Dragonfly.  Keena  used  this  tradespace  environment  to  run  trials  with  18 
vehicle  variants,  each  of  which  had  binary  configuration  options  for  consideration.  Keena 
(2011)  amassed  data  from  14  participants,  running  15  different  missions,  on  10  randomly 
assigned  vehicle  variants  totaling  approximately  100  missions  per  vehicle.  The  binary 
configuration  options  he  used  were 

•  tracked  versus  wheeled, 

•  levels  of  survivability  (two  levels), 

•  levels  of  lethality  (two  levels), 

•  levels  of  mobility  (two  levels),  and 

•  two  training  vehicles. 
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Through  this  effort,  Keena  (2011)  gained  insights  that  support  further  efforts  for 
ESP  by  looking  at  the  vehiele’s  mobility,  lethality,  and  survivability  in  eoncert.  The 
environment  offered  useful  tradespaee  feedbaek  based  on  the  binary  input  options  as  well 
as  output  metrics  on  mission  success/failure,  mission  time,  and  vehicle  health.  The  virtual 
environment  offers  a  superior  evaluation  environment  in  that  there  was  no  degradation  in 
operator  performance  from  fatigue  (Keena,  2011).  From  the  warfighter  perspective,  the 
reduction  of  fatigue  affords  each  alternative  being  considered  a  more  consistent 
evaluation  environment. 

4,  Synthetic  Environments  for  Assessment 

Rudolf  Darken  and  Joseph  Cohn  (2012)  sought  “nonlinear  innovation”  through 
Synthetic  Environments  for  Assessment  (SEA)  to  optimally  support  achievement  of  a 
leap-ahead  solution  that  could  disrupt  the  status  quo  and  propel  warfighters  to  a  clear 
advantage.  SEA  has  many  of  the  same  objectives  as  ESP  but  is  focused  on  human 
systems  (man-in-the-loop)  specifically  investigating  the  trade-offs  between  users 
(including  teams)  and  equipment.  SEA  calls  for  “calibrated  scenarios”  that  are  validated 
and  realistic  to  be  used  for  testing  ideas.  It  also  identifies  the  need  for  reusable 
components  (models  and  software)  to  ensure  repeatability  in  assessment  (Darken  & 
Cohn,  2012).  These  concepts  are  likely  to  enhance  ESP  in  the  future. 

5.  System  Engineering  2025 

System  Engineering  (SE)  2025,  as  proposed  by  Robert  Smith  and  Brian  Vogt 
(2014),  could  incorporate  ESP  at  the  front  end  of  a  process  that  aims  to  provide  flexible 
and  adaptable  solutions  for  rapid  fielding  even  at  austere  locations  by  leveraging 
techniques  such  as  additive  manufacturing.  This  approach  on  “mass  customization”  is 
based  on  enabling  a  platform  in  which  soldiers  and  engineers  can  collaborate  on 
technology  solutions  as  well  as  the  tactics  to  employ  the  technology.  SE  2025  seeks  to  be 
an  enabling  technology  in  its  own  right,  supporting  warfighter  needs  in  a  more  rapid 
fashion  (Smith  &  Vogt,  2014). 
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6,  Framework  for  Assessing  Cost  and  Technology 

A  tool  currently  employed  by  the  Marine  Corps  is  the  Framework  for  Assessing 
Cost  and  Teehnology  (FACT),  which  incorporates  simulation  into  the  life-eycle 
management  of  a  system  (O’Neal,  2011).  There  are  extensions  in  development  to  bring 
FACT  into  a  simulated  seenario  environment  for  rudimentary  evaluation  of  features 
given  speeific  eonditions  and  mission  demands  (Ender,  2014).  To  facilitate  the  evaluation 
of  trades  between  capabilities  and  available  resources,  tools  are  required  to  visualize 
information.  FACT  is  a  highly  refined  and  capable  design  trade-off  analysis  environment 
that  effectively  meets  these  goals  for  evaluation  and  visualization  (Velazquez, 
2014).  FACT  offers  an  open-arehitecture  web  service  to  provide  rapid  exploration  of 
engineering  design  trade-offs  (Yates,  2012).  Performance,  reliability,  risk,  and  cost  over 
the  life  cycle  of  a  system  ean  be  explored  in  this  comprehensive,  near-real  time, 
government-owned  resource  built  for  the  Marine  Corps  (O’Neal,  2011). 

The  detail,  fidelity,  and  physieally  based  aecuracy  presented  in  the  FACT 
environment  far  surpasses  what  is  envisioned  for  ESP.  Rather,  ESP  is  a  precursor  where 
design  ideas  are  aeeumulated,  vetted,  and  aeeepted  or  discarded  based  on  estimates  of 
performance.  PACT  is  a  tool  to  hone  a  good  design  into  a  better  design.  ESP  is  a  tool  to 
help  find  the  good  design  in  the  first  place.  Alternately,  refined  solutions  developed  and 
assembled  in  the  PACT  library  could  be  made  available  for  emulation  in  ESP  (in  an 
iterative  design  proeess)  to  assess  performanee  in  a  speeifie  operational  environment  or  a 
novel  tactieal  employment  seenario.  In  conjunction  with  PACT,  ESP  provides  the 
opportunity  to  demonstrate  virtual  employment  of  a  combat  system  in  a  combat  scenario 
while  garnering  contributions  from  a  wide  breadth  of  experienced  users. 

7,  Executable  Architecture  Systems  Engineering 

Executable  Arehitecture  Systems  Engineering  (EASE)  is  a  collaborative 
environment  in  which  to  assess  the  detailed  systems  concepts  that  are  developed  in  PACT 
(see  Pigure  9).  Beyond  the  feature  and  functionality  tradespace  offered  by  PACT,  EASE 
provides  the  opportunity  to  examine  measures  of  effeetiveness  and  provide  results  that 
ean  be  sent  baek  to  PACT  for  further  evaluation  and  “tradestudy”  (Ender,  2014,  p.  3). 
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The  enterprise  architeeture  established  between  FACT  and  EASE  is  similar  to  ESP  but 
does  not  eontain  the  critieal  inelusion  of  input  and  participation  from  a  variety  of  subject 
matter  experts  (SMEs)  across  the  spectrum  of  military  operations,  found  by  using  a 
crowdsourcing  technique  to  solicit  and  vet  input  from  a  much  broader  variety  of 
contributors  than  the  science  and  technology  contributors  who  would  automatically  be 
involved  in  critical  coordination  and  decisions  on  a  program  of  record. 
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Eigure  9.  EASE  User-Eevel  Interfaces  (from  Eesinski,  McCarthy,  & 

Gaughan,  2013) 


D.  GAME  ANALYTICS 

Measurement  within  games,  or  game  analytics,  is  a  way  to  glean  useful 
information  from  game  play  that  players  may  or  may  not  be  aware  is  being  collected. 
Games  provide  an  inclusive  way  to  accumulate  input  and  feedback  from  participants 
without  creating  an  unnecessary  burden.  Games  can  invite  creativity  and  encourage 
responses  from  those  who  might  not  otherwise  participate.  Eike  any  mode  of 
entertainment,  video  games  need  an  audience  to  maintain  viability.  The  potential  for 
profit  has  led  game  developers  to  take  a  closer  look  at  what  they  are  developing,  how  it  is 
being  developed,  and  what  aspects  gamer  the  most  loyalty,  reinforcement,  and 
profitability  from  their  player  population  (Seif  el  Nasr,  2013). 

The  astronomic  growth  of  the  video  game  industry  has  motivated  developers  and 
publishers  to  seek  an  edge  in  determining  the  makings  of  a  great  game  as  compared  to  an 
ordinary  game.  Game  analytics  is  a  burgeoning  field  of  study  that  addresses  this  need.  In 
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a  comprehensive  eolleetion  of  work  on  game  analyties  and  its  applieation,  Seif  El-Nasr, 
Draehen,  and  Canossa  (2013)  provide  a  definition  of  game  analyties,  whieh  they  deseribe 
as 

the  proeess  of  discovering  and  eommunicating  patterns  in  data,  towards 
solving  problems  in  business  or  eonversely  predictions  for  supporting 
enterprise  deeision  management,  driving  aetion  and/or  improving 
performanee.  Game  analyties  is  a  speeifie  applieation  domain  of  analyties, 
deseribing  it  as  applied  in  the  eontext  of  game  development  and  game 
researeh.  (pp.  14-15) 

Seif  El-Nasr  et  ah,  further  breaks  the  definition  of  game  analyties  into  two  distinct 
segments:  the  “game  as  a  produet”  that  should  provide  a  good  user  experienee,  and  the 
“game  as  a  project”  (p.  15)  that  should  perform  well  on  its  own  and  in  comparison  to 
other  games.  ESP  is  eoneerned  with  both  produet  and  projeet:  eontinuous  interest  from 
players  and  a  well-developed  game  environment  eapable  of  handling  the  desired  user 
load  and  suffieient  instrumentation.  In  ESP,  akin  to  the  game  developers,  S&T  experts 
working  with  a  program  offiee  ean  advise  the  program  manager  about  what  data  to 
eolleet  and  how  to  eorrelate  that  data  effeetively  to  inform  deeisions  for  the  program. 

Game  developers  intend  to  eraft  games  that  aehieve  eommereial  sueeess.  That 
eommereial  sueeess  is  based  on  a  positive  user  experienee.  The  ability  to  observe  player 
aetions  and  reaetions  in  the  game  environment  offers  the  opportunity  to  dynamieally 
update  a  game  to  maximize  the  monetization  of  players.  Game  analyties  ean  be  applied  to 
inform  deeision-making  by  providing  a  myriad  of  information  to  supplement  other 
available  business  intelligenee  data.  Games  are  not  always  played  on-site  for  developer 
observation;  therefore,  telemetry  (data  obtained  over  distance)  provides  a  view  into  a 
player’s  decisions  and  resulting  sueeess,  failure,  and  game  behavior  (Seif  El-Nasr  et  ah, 
2013,  p.  16).  Game  metrics  are  interpretable  measures  of  something  related  to  games — 
quantitative  measures  of  attributes  of  objeets  within  the  game  environment.  Mierosoft 
uses  Xbox  Eive  data  from  18,000  volunteer  players  to  eapture  games  played, 
aehievements  earned,  and  inferred  presenee  information.  The  data  eomes  through  an 
XME  feed  from  eaeh  player’s  Xbox.  Mierosoft  also  mines  online  eommunities  for 
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threads  relevant  to  games  under  development  (Zimmermann,  Phillips,  Nagappan,  & 
Harrison,  2012). 

Within  the  first-person  shooter  genre,  metrics  that  might  apply  to  ESP  include 
“weapon  use,  trajectory,  item/asset  use  ...  loss/win,  heat  maps,  team  scores  ...  vehicle 
use  metrics,  strategic  point  captures/losses  ...  avatar  movement  and  posture,  ...  AI- 
enemy  damage  inflicted,  and  projectile  traces”  (Seif  El-Nasr  et  al.,  2013,  p.  25).  None  of 
these  metrics  require  special  instrumentation  or  hardware.  Players  are  unaware  that  data 
is  being  collected.  Gleaning  useful  strategic  information  is  more  difficult.  Parameters  that 
may  apply  to  strategies  employed  may  include  monitoring  the  frequency  of  the 
previously  mentioned  attributes,  such  as  event  frequency  and  events  initiated. 

E.  REACHING  THE  CROWD 

Crowdsourcing  in  the  context  of  ESP  is  the  practice  of  obtaining  ideas  by 
soliciting  contributions  from  a  large  body  of  players.  Gaining  access  to  a  significant 
volume  of  information  without  negatively  impacting  the  body  providing  that  information 
is  a  worthy  goal.  Consequently,  game  analytics  that  are  transparent  to  the  players  are 
essential.  Crowdsourcing  is  employed  in  such  venues  as  massively  multi-player  online 
games  (MMOGs)  where  troves  of  data  on  character/player  activities  can  be  collected, 
parsed,  and  evaluated  for  relevance.  The  Entertainment  Software  Association  (ESA)  2013 
report  of  sales,  demographics,  and  usage  data  stated  that  “62%  of  gamers  play  games 
with  others  either  in  person  or  online  and  77%  of  gamers  who  play  with  others  do  so  at 
least  one  hour  per  week”  (p.  5).  At  comparable  rates,  the  amount  of  data  that  could  be 
captured  from  several  hundred  warfighter  players  is  substantial. 

The  ESP  framework  will  need  to  be  easily  accessible  via  the  internet  to  allow 
players  across  the  country  to  access  the  game  environment  off  duty,  or  even  on  duty  if  it 
is  deemed  appropriate  or  necessary.  Keeping  players  coming  back  to  the  game 
environment  to  play  a  variety  of  systems  in  multiple  scenarios  offers  the  ability  to 
longitudinally  track  their  valuable  inputs  and  progression  through  the  game  environment, 
which  is  a  necessary  requirement  for  ESP  to  succeed. 
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F. 


LIMITATIONS  OF  GAMES 


By  its  definition,  a  game  is  “a  physieal  or  mental  aetivity  or  eontest  that  has  rules 
and  that  people  do  for  pleasure”  (“Game,”  2014).  This  definition,  at  faee  value,  would  not 
lead  one  to  believe  that  a  game  eould  be  useful  for  praetieal  purposes.  Finding  a  balance 
between  the  generally  accepted  principle  that  games  are  for  pleasure  or  entertainment 
rather  than  for  the  development  of  functional  outcomes  is  critical  to  employment  of  a 
game  environment  for  the  purpose  of  conducting  a  practical  simulation  in  support  of 
acquisition.  Commercial  video  game  developers  use  game  analytics  in  part  to  aid  the 
creation  of  more  entertaining  games  to  build  a  strong  customer  base.  Therefore,  ESP  will 
be  competing  against  countless  computer  and  console  game  options  for  the  time  of  the 
warfighters.  Keeping  players  engaged  and  returning  to  play  in  ESP  will  be  a  challenge. 

As  games  become  more  complicated,  the  cost  of  developing  and  running  those 
games  will  increase.  Cost  can  be  interpreted  as  pay  for  a  programmer  to  build  a  more 
realistic,  higher  fidelity  model,  the  cost  of  a  more  robust  platform  in  which  to  host  the 
more  complex  models  and  scenarios,  and  also  the  computational  cost  of  rendering  on 
screen  and  communicating  between  distributed  players  and  the  game  server.  If  the  game 
fidelity  is  made  too  complex,  individuals  using  less  capable  resources,  such  a  low-cost 
personal  laptop,  may  not  be  able  to  play  the  scenarios  at  a  satisfying  level  in  a  real-time 
online  environment.  Given  the  multitude  of  competitor  game-play  options,  such  as 
smartphone  applications  or  console  games,  unsatisfied  players  may  elect  to  spend  their 
time  doing  things  other  than  playing  ESP  scenarios.  The  counter  for  ESP  is  the  altruistic 
proposition  that  the  warfighters  engaging  in  game  play  have  a  vested  interest  in  the 
successful  development  of  future  DOD  systems. 

A  game  environment  may  not  be  able  to  convey  robustly  some  challenging 
aspects  of  system  development,  such  as  logistics.  Care  must  be  taken  to  consider 
logistical  realities  that  may  be  elusive  in  a  game  environment.  Damage  to  a  system,  the 
need  to  conduct  preventive  and  corrective  maintenance,  and  the  requirement  to  recharge 
or  refuel  assets  prior  to  or  during  employment  are  a  few  examples  that  would  be 
challenging  to  represent  in  a  game  environment.  In  short,  logistical  concerns  can  be 

modeled  but  might  lack  detail  to  keep  players  engaged  in  real  time  or  produce  reliable 
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outcomes.  Much  as  actual  final  validations  of  system  models  cannot  be  accomplished  in 
the  virtual  realm  of  a  game,  challenging  concepts,  such  as  logistics,  will  need  follow-on 
evaluation  in  a  more  robust,  higher  fidelity  environment.  ESP  is  proposed  for  early  stages 
of  evaluation  and  development  where  a  multitude  of  options  can  be  considered. 

Despite  the  potential  limitations,  the  ubiquity  of  games  is  evident  in  that  they  are 
played  by  58%  of  Americans,  and  51%  of  American  households  have  at  least  one  game 
console  (ESA,  2013).  The  pervasive  presence  of  video  games  makes  the  game 
environment  a  promising  venue  in  which  to  communicate  with  and  collect  data  from  the 
crowd  of  warfighters  across  the  DOD  in  the  early  stages  of  concept  development. 

G.  DOD  DECISION  SUPPORT 

There  is  a  strong  juxtaposition  between  the  formal,  highly  organized  process  map 
of  the  DOD  decision  support  framework  and  the  malleable  platform  that  can  be 
developed  in  a  game  environment.  The  game  environment  offers  a  venue  that  widely 
engages  a  principle  stakeholder — the  end  user — without  the  intimidation  that  can  occur 
when  dealing  with  the  processes  related  to  the  JCIDS,  DAS,  and  PPBE.  Crowdsourcing 
allows  for  contact  with  and  extraction  of  critical  information  from  a  wide  population  of 
users.  Crowdsourcing  within  ESP  also  solicits  the  expertise  offered  by  S&T  professionals 
seeking  out  profound  technology  developments.  This  objective  is  not  modest,  but 
meeting  it  will  help  the  United  States  maintain  its  technological  dominance  on  the 
battlefield. 
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III.  EARLY  SYNTHETIC  PROTOTYPING  CONCEPT 


ESP  offers  an  opportunity  for  exploring  proposed  systems  and  system 
characteristies  in  the  low-cost  tradespace  of  a  virtual  environment.  Using  a  game 
environment  as  a  platform  is  a  way  to  reach  out  to  a  broad  population  of  users  to  gain 
input  from  those  with  either  current  operational  experience,  technological  savvy,  or  both. 
Within  the  game  environment,  the  warfighters  can  provide  input,  feedback,  and  proposals 
toward  systems  and  methods  of  tactical  employment  bringing  ESP. 

A.  THE  CONCEPT 

ESP  is  a  process  and  set  of  tools  that  enables  warfighters  to  inform  technology 
development  and  acquisition  decisions  by  assessing  emerging  technologies  in  a  game 
environment.  Collaborators  in  acquisition,  S&T,  and  industry  can  develop  models  and 
scenarios  for  play  and  assessment.  ESP  allows  an  unbounded  increase  in  potentially 
disruptive  ideas  to  be  explored  at  minimal  cost  by  inviting  warfighters  at  all  levels  to 
drive,  define,  and  refine  future  systems. 

The  DOD  needs  a  means  to  complement  or  reinforce  the  competitive  prototyping 
effort  and  ensure  that  requirements  and  evaluation  criteria  are  clear  prior  to  soliciting  for 
physical  prototypes.  Eigure  10  depicts  hypothetical  cost  and  value  curves  based  on 
increasing  fidelity  of  prototype  medium.  The  green  arrow  shows  where  ESP  will  seek  to 
exploit  the  gap  between  preliminary  sketches  and  high  fidelity  simulations,  such  as 
PACT  and  EASE,  using  a  game  environment.  The  relative  gain  in  value  and  relative 
increase  in  cost  of  a  game  environment  over  pencil  sketches  is  better  than  what  could  be 
achieved  by  skipping  to  a  high  fidelity  model  and  simulation.  Each  increasing  level  of 
prototype  fidelity  provides  some  overlapping  as  well  as  distinct  opportunities  to  develop 
insights.  Ultimately,  the  closer  the  prototype  is  to  the  actual  production  system,  the  higher 
the  value  and  level  and  of  insight  researchers  and  future  users  will  have  into  the  system, 
and  how  that  system  can  be  effectively  employed. 
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Prototype  Fidelity 

Figure  10.  Hypothetical  Cost  and  Value  Curves  Based  on  Increasing 

Prototype  Fidelity  (from  Murray,  Darken,  Vogt,  &  Goerger,  2014) 

ESP  can  be  accomplished  in  support  of  successful  acquisition  of  modern  and 
effective  programs  of  record.  The  ESP  framework  will  be  capable  of  providing  early, 
cost-effective  feedback  from  an  operationally  experienced  source.  The  ideal  point  at 
which  to  insert  ESP  is  during  concept  development.  ESP  could  contribute  the  ability  to 
evaluate  multiple  system  variants  and  scenarios  in  support  of  the  requirements 
development  phase  of  the  acquisition  cycle.  Employing  ESP  has  the  potential  to  develop 
a  strong  foundation  on  which  to  proceed  with  acquisition  on  any  scale.  Prom  this  vantage, 
ESP  can  become  the  cornerstone  of  the  greater  effort  toward  Engineering  Resilient 
Systems  (ERS). 

1,  Early  Synthetic  Prototyping:  A  Cornerstone  of  Engineering  Resilient 
Systems 

ESP  is  not  a  holistic  solution,  but  rather  a  concept  that  fits  into  the  greater 
acquisition  cycle.  More  specifically,  ESP  will  become  a  cornerstone  of  ERS,  which  is 
intended  to  address  the  entire  acquisition  process,  not  limited  to  the  early  concept  phases. 
(See  Appendix  B  for  a  depiction  of  ERS  interaction  with  the  acquisition  cycle.)  ERS 
seeks  to  enable  better  acquisition  decisions  by  providing  a  rigorous  science  and 
engineering  process  on  an  open  framework  for  requirements  generation,  analysis  of 
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alternatives,  and  prediction  of  life-cycle  performance  and  costs  (Goerger,  2014).  ESP 
should  be  considered  one  facet  of  what  ERS  will  eventually  become. 


The  assistant  secretary  of  defense  for  research  and  engineering  (ASD[R&E])  has 
made  ERS  a  priority  to  focus  on  improving  engineering,  design,  and  development  of 
defense  systems  through  targeted  application  of  science  and  technology.  Eigure  11 
depicts  how  ERS  seeks  to  link  warfighter  needs  across  common  platforms,  such  as 
reliability  and  sustainability,  with  S&T  resources  to  collaborate  toward  producing  a 
robust  portfolio  of  rapid,  reconfigurable  systems.  As  described  by  Tommer  Ender  (2014), 
ERS  aims  to  accomplish  development  of,  among  other  aims, 

•  “more  complete  and  robust  requirements”  supported  by  an  increase  in  the 
parameters  and  scenarios  used  to  help  set  those  requirements  (p.  4), 

•  a  quantified  adaptability  to  a  changing  mission  (p.  4),  and 

•  “reduction  in  time  to  complete  systems  by  reducing  rework”  (p.  4  ). 
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Figure  11.  Engineering  Resilient  Systems  Concept  (from  Goerger,  2014) 
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2,  Early  Synthetic  Prototyping:  Sources  for  Innovation 

The  DOD  has  often  looked,  with  limited  success,  outside  the  defense 
establishment  for  potentially  disruptive  change.  The  private  sector  is  not  well  tuned  to 
understand  modern  warfare  and  all  the  subtleties  that  differentiate  a  revolutionary  idea 
from  yet  another  evolutionary  improvement.  In  the  July  2014  edition  of  Army  Magazine, 
Hill  and  Allen  proposed  that  innovation  in  a  military  context  can  be  accomplished 
through  “brilliant  mistakes”  (p.  30)  and  “anomaly-seeking  behavior”  (p.  28).  Further, 
they  offered  that  “in  war,  a  gap  between  expected  and  actual  performance  of  a  plan, 
tactic,  or  system  creates  a  demand  signal  for  change”  (Hill  &  Allen,  2014,  p.  28).  The 
creativity  and  innovative  spirit  for  disruptive  change  exists  in  each  service  member,  but  it 
remains  a  largely  untapped  resource.  ESP  can  provide  an  opportunity  to  harness  that 
resource,  allowing  for  the  free  flow  of  ideas  to  both  identify  and  assess  capabilities  and 
tactical  employment  concepts  early  in  the  design  phase.  At  this  point  in  the  acquisition 
cycle,  the  cost  of  change  is  lower,  and  making  changes  to  requirements  and  specifications 
does  not  have  voluminous  second-  and  third-order  effects.  ESP  offers  a  low-cost,  flexible 
tool  to  augment  requirements  development  through  a  medium  familiar  to  warfighters — 
gaming. 

ESP  uses  a  distributed  game  network  as  the  medium  to  open  a  conduit  for  the 
flow  of  ideas.  Rather  than  relying  exclusively  on  expert  designers  within  acquisition 
programs  to  generate  a  small  number  of  good  ideas  that  are  then  prototyped,  tested,  and 
revised  (at  significant  cost  and  time),  ESP  facilitates  the  development  of  an  unbounded 
number  of  design  options  in  the  concept  phase.  Those  options  are  tested  and  assessed  as 
virtual  prototypes  in  a  game  network.  Warfighter  players  “play”  the  virtual  systems  while 
analysts  gather  data  via  game  analytics  to  identify  what  works  and  what  does  not. 

ESP  proposes  the  use  of  a  crowdsourcing  technique  to  assess  the  utility  and 
efficiency  of  virtual  prototypes.  Crowdsourcing  works  only  if  the  number  of  players  is 
very  large.  With  a  large  crowd  of  collaborators,  no  single  player  is  assumed  to  be  an 
expert  in  design  or  assessment.  Rather,  as  a  group,  the  crowd  is  capable  of  contributing  to 
the  design  process  (Surowiecki,  2005).  ESP  players  could  be  anyone  from  a  private  fresh 

out  of  boot  camp  to  an  experienced  combat  veteran.  The  assumption  stands  that  all 
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contributors  have  something  interesting  to  offer.  Therefore,  ESP  does  not  eoneem  itself 
with  weeding  out  bad  designs;  players  do  that.  The  purpose  of  ESP  is  not  to  find  the 
perfeet  design;  it  is  to  find  the  best  possible  subset  of  virtual  prototypes  for  aequisition 
professionals,  such  as  program  managers,  to  pursue  further  hand  in  hand  with  their 
scienee  and  technology  partners.  The  pursuit  ean  be  eondueted  with  eonfidenee  that  one 
of  the  ideas  is,  at  best,  a  game  ehanger,  and,  at  worst,  a  praetieal  solution,  based  on 
evidenee  displayed  through  ESP.  Additionally,  ESP  offers  the  opportunity  to  explore 
foree  design  and  foree  employment  in  eonjunetion  with  eapability  development  at  the 
operator  and  small-unit  levels.  The  ability  to  observe  taetieal  employment  of  a  system 
without  going  to  the  expense  of  engaging  troops  and  resourees  in  a  physieal  exereise 
offers  the  opportunity  to  explore  far  more  options  and  then  to  highlight  whieh  options 
deserve  further  review. 

B,  THE  BENEFIT 

ESP  is  a  persistent  game  network  with  instrumented  seenarios  and  metries  for 
exploring  alternative  future  designs  to  inform  present  deeision-making.  The  stakeholders 
inelude  warfighters  and  aequisition  professionals  in  government  and  industry,  and 
seienee  and  technology  programs.  Eigure  12  identifies  the  prineiple  benefit  of  the 
ineorporation  of  ESP,  moving  the  ehange  demand  from  its  eurrent  position  late  in  the 
aequisition  eyele  to  a  mueh  earlier  position  where  the  eost  of  ehange  is  more  reasonable 
and  ean  be  readily  adapted  into  the  program’s  budget  and  sehedule.  With  a  reduetion  in 
costly  late-stage  changes,  eustomers  throughout  the  DOD  will  be  able  to  employ  the  right 
systems  in  a  mueh  more  efficient  manner,  redueing  the  need  to  unneeessarily  refit 
systems  after  production. 
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Figure  12.  Shift  of  Change  Demand  With  ESP  (from  Murray  et  ah,  2014) 


ESP  builds  on  ideas  and  concepts  outside  of  the  military.  Eor  example,  Wang  et 
al.  (2008)  entertained  the  notion  that  a  synthetic  environment  could  be  used  in  industrial 
product  design  as  a  tool  to  communicate  concepts  for  “traditional  prototype  evaluation,  or 
collaborative,  interactive,  user-centered  dynamic  prototyping”  (p.  3).  Beyond  prototype 
development  and  evaluation,  Wang  et  al.  (2008)  proposed  scenario-based  design  (SBD) 
to  allow  participants  to  adapt  a  scenario  to  further  communication  of  the  individual’s 
perspective  of  the  concept  under  development.  ESP  builds  on  these  ideas  and  brings  it 
into  the  DOD  acquisition  framework. 

Referring  to  the  ESP  schematic  in  Eigure  13  (see  Appendix  C  for  a  larger  image), 
acquisition  professionals  (1)  use  scenario  editing  tools  to  develop  concept  ideas  for 
testing.  These  are  playable  scenarios  (2)  that  are  instrumented  with  metrics  of  interest 
(e.g.,  system  selected,  rounds  fired,  speed  attained,  distance  traveled)  specified  by  the 
program  office.  These  scenarios  are  made  available  for  play  via  a  game  server  (3)  that 
allows  distributed  warfighter  players  (4)  to  play,  capturing  diagnostic  information. 
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Communities  of  players  are  coordinated  via  conventional  social  media  (5).  Players  can 
modify  scenarios  locally.  ESP  does  not  assume  the  best  scenarios  will  come  from  the 
program  office.  More  likely,  the  best  ideas  will  be  modifications  of  those  ideas  made  by 
players.  Game-play  diagnostic  data  (6)  and  modified  scenarios  (7)  are  returned  to  the 
server  and  then  to  the  program  office  (8).  Program  offices  can  also  interact  with  players 
via  the  community  (9).  Science  and  technology  programs  (10)  will  use  ESP  to  test  new 
ideas  at  minimal  expense  before  follow-on  larger  investment.  They  will  also  use  ESP  to 
demonstrate  concepts  to  potential  transition  customers  (11)  to  obtain  buy-in  early,  which 
will  facilitate  an  increase  in  successful  transition  to  programs  of  record. 
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Figure  13.  ESP  Schematic  (from  Murray  et  ah,  2014) 


C.  THE  BARRIERS 

The  ESP  concept  has  great  face  validity  as  a  way  to  explore  new  ideas.  But 
presently,  it  is  only  a  concept.  A  number  of  critical  steps  must  be  taken  before  it  is 
realized,  specifically  as  outlined  in  Table  1. 
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Table  1.  Critical  Steps  for  Early  Synthetic  Prototyping 


The  network  must  be  distributed 
worldwide  and  playable  by  any 
authorized  player. 

The  game  America’s  Army  accomplished  this 
over  10  years  ago  (Zyda,  Mayberry,  Wardynski, 
Shilling,  &  Davis,  2003).  This  is  a  solved 
problem,  but  managing  specialized  player 
groups  will  be  a  challenge. 

Scenario  editing  must  be  simple  yet 
expressive. 

Scenario  editing  for  a  program  office  is  similar 
to  level  editing  in  game  design.  A  key  issue  is 
setting  data  collection  triggers  to  capture 
relevant  play  data. 

Game  play  must  be  easy  and 
entertaining  or  players  will  simply 
spend  their  time  doing  something 
else. 

America’s  Army  dealt  with  this  as  well.  Just 
because  a  game  is  free  does  not  mean  players 
will  play.  Rather  than  competing  for  players’ 
money,  ESP  will  compete  for  their  time.  Players 
should  also  know  that  the  Army  is  listening  to 
what  they  have  to  say. 

Players  must  be  able  to  easily 
modify  scenarios  to  create  new 
designs  and  configurations. 

Although  simpler  than  a  full-level  editor,  players 
must  be  able  to  easily  modify  content  similar  in 
complexity  to  the  sand  table  development 
environment  on  the  site  Garry’s  Mod 
(http://www.garrysmod.com). 

Gleaning  useful  information  from 
game  play  must  be  simple,  or  better 
yet,  transparent  to  players. 

There  are  two  inherent  questions  here:  (1)  can 
information  be  collected  from  game  play?  and 
(2)  is  it  useful  to  decision-makers? 

Chapter  IV  details  the  study  we  conducted  at  the  Naval  Postgraduate  School 
(NPS)  focusing  on  the  last  element  in  Table  1:  Can  information  be  collected  from  game 
play,  and  is  it  useful?  Along  with  my  fellow  researchers,  I  sought  to  determine  whether  it 
was  possible  to  glean  useful  information  from  game  play  and  whether  the  information 
collected  was  useful  to  those  groups  responsible  for  the  development  of  technology 
solutions  and  systems. 

D,  OPTIONS  FOR  CONSIDERATION 

In  addition  to  the  distributed  game  network  allowing  warfighters  to  play  during 
their  off-duty  time,  it  is  worthwhile  to  consider  that  there  may  be  situations  and 
opportunities  where  evaluation  within  a  game  in  a  locally  controlled  environment  may  be 
useful.  Despite  the  proposed  robust  support  for  a  distributed  game  environment,  the  ESP 

framework  could  support  operations  in  a  more  controlled  environment,  such  as  a  training 

34 


or  simulation  center.  For  this,  a  hierarchical  concept  is  proposed  where  early  assessments 
are  considered  by  the  largest  possible  crowd  (see  Figure  14).  The  community  invited  to 
partake  is  then  progressively  reduced  to  include  more  subject  matter  or  design  experts 
while  particular  areas  are  being  evaluated.  For  example,  SMEs  may  have  a  unique 
perspective  on  the  intricacies  required  for  a  control  interface  for  a  system.  SMEs  may 
also  need  to  address  properties  of  the  virtual  models  that  require  more  rigorous  evaluation 
from  a  physics  perspective.  The  collection  of  player  data  and  feedback  could  be 
combined  with  in-person  feedback  provided  via  after-action  sessions,  much  like  the  post¬ 
play  interviews  conducted  in  our  study  at  NFS.  For  an  established  program  of  record,  the 
opportunity  to  get  first-hand  feedback  from  players  immersed  in  the  proposed  technology 
can  only  complement  the  data  points  that  can  be  collected  and  analyzed  in  each  scenario. 


Broad  Engagement  service-wide 

n-tiers  of  progressively  smaller 
community  of  contributors 

SME  Engagement 
(distributed) 

SME 

Engagement 

(local) 

Early  Synthetic  Prototyping  (ESP)  Network  Hierarchy 

Figure  14.  Proposed  Network  Hierarchy 


E.  MINIMUM  REQUIREMENTS:  A  WORK  IN  PROGRESS 

The  Army  has  begun  to  develop  a  detailed  specification  for  the  ESP  framework. 

The  following  requirements  have  been  discussed  and  roughly  prioritized  to  date  and  are 

subject  to  modification  and  update  as  the  efforts  for  ESP  continue  into  a  pilot  and  full 

study  in  FY2015  (B.  Vogt,  personal  communication,  June  13,  2014). 

•  Availability  to  warfighters  on  or  off  duty  hosted  on  a  server  accessible  via 
web  browser. 


35 


•  Automated  data  collection  of  game  play  for  automated  and  manual 
analysis.  Do  not  annoy  the  player  with  burdensome  data  collection 
techniques  such  as  surveys  or  questionnaires. 

•  Technical  maturity  of  both  the  game  environment  and  the  visual  content  to 
ensure  a  quality  user  experience  from  connectivity,  log-in,  and  game  play, 
to  vibrancy  of  models  and  terrain.  Note  that  entertainment  value  need  not 
outweigh  the  observation  of  reasonable  outcomes  of  in-game  interactions. 

•  Commonality  and  compatibility  with  the  larger  modeling  and  simulation 
community  to  included  achieving  distributed  interactive  simulation  (DIS) 
compliance. 

•  Ability  to  model  future  (unknown)  capabilities. 

•  Balance  of  government  ownership  with  vendor  proprietary  material. 
Ideally,  an  innovative  vendor  will  allow  government  access  to  make 
modifications  when  necessary  and/or  prioritize  efforts  without  exorbitant 
fees. 

•  Incorporation  of  existing  government-owned  models,  such  as  those  used  in 
Virtual  Battlespace  Simulation  Version  2  or  3  (VBS2  and  VBS3). 

•  Connection  to  a  readily  available  and  monitored  discussion  forum.  This 
allows  players  to  engage  with  each  other,  and  developers  to  respond  to 
requests  for  change  or  updates  to  prototypes  and  scenarios  to  keep  them 
relevant  and  offer  branches  for  evaluation. 

•  Audio  connectivity  to  allow  for  real-time  interaction  between  distributed 
players. 

•  The  fidelity  of  physics  must  mimic  reality  but  does  not  need  to  be  so 
robust  that  it  inhibits  system  performance. 

•  Distribution  of  players  in  small  groups  of  four  to  six  players  initially  with 
eventual  opportunity  to  scale  upward  for  larger  engagement  scenarios. 

•  User  interface  for  ESP  should  initially  support  standard  input  output 
devices,  such  as  keyboard  and  mouse  with  acceptable  visibility  on  the 
monitor  of  a  desktop  or  screen  of  a  laptop.  Incorporation  of  functionality 
for  gamepads  is  preferred  to  retain  buy-in  from  players  familiar  with 
gaming.  Future  development  for  tablet  or  smartphone  may  be  considered 
if  the  system  under  evaluation  could  be  supported  in  those  environments. 
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F. 


THE  WAY  AHEAD 


As  the  Army  continues  development  of  ESP,  the  minimum  requirements  will  be 
refined  to  meet  the  needs  of  this  avant-guard  program.  With  a  goal  for  limited  access  and 
distribution  in  early  calendar  year  2015,  ESP  must  take  care  not  to  discourage  early 
adopters  with  an  incomplete  solution.  The  study  conducted  in  support  of  this  research 
effort  supports  the  establishment  of  a  robust  ESP  framework.  The  study  sought  to 
determine  whether  the  game  environment  proposed  for  ESP  could  meet  the  needs  of  the 
DOD  as  it  seeks  out  revolutionary  technology  solutions  to  meet  warfighter  needs. 
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IV.  STUDY  METHOD,  RESULTS,  AND  ANALYSIS 


We  conducted  a  study  at  the  Naval  Postgraduate  School  (NPS)  at  the  Modeling, 
Virtual  Environments,  and  Simulations  Institute  (MOVES).  In  the  study,  we  simulated  an 
ESP-like  environment  where  a  concept  robotic  vehicle  was  placed  in  a  series  of  realistic 
combat  scenarios  and  players  were  asked  to  use  the  vehicle  to  meet  scenario  objectives. 
Of  particular  interest  were  how  the  players  would  use  the  vehicle,  what  form  their 
feedback  might  take,  and  whether  their  feedback  would  be  useful  to  a  program  office 
responsible  for  making  critical  decisions  on  design  and  employment. 

Our  ESP  study  addressed  what  kind  of  information  can  be  obtained  from 
warfighters  playing  games  to  assess  a  novel  technology  solution  in  a  virtual  environment 
and  ultimately,  whether  that  information  will  be  useful  to  the  development  of  future 
capabilities.  The  study  attempted  to  validate  the  employment  of  games  as  a  tool  for 
assessment  of  a  proposed  system,  as  well  as  rudimentary  development  and  analysis  of 
alternatives.  The  Army  Capabilities  Integration  Center  (ARCIC)  in  conjunction  with  the 
Army  Tank  and  Automotive  Research,  Development,  and  Engineering  Center 
(TARDEC)  provided  a  novel  technology  for  evaluation  in  a  game  environment  in  the 
form  of  the  robotic  Wingman.  System  models  were  developed  in  the  military  first-person 
tactical  simulation  environment  of  Virtual  Battlespace  (VBS2)  to  examine  several 
variants  of  Wingman  in  the  context  of  three  distinct  employment  scenarios  within  the 
game  environment.  The  specific  questions  addressed  in  the  study  were 

•  Can  information  be  collected  from  warfighters  playing  with  novel 
technology  concepts  in  a  game  environment? 

•  Is  that  information  useful  to  the  development  of  the  system  or  its  tactical 
employment? 

A.  METHOD 

The  Army  is  exploring  a  semi-autonomous  robotic  vehicle  concept  called 
Wingman.  There  are  many  ideas  for  how  Wingman  might  be  used  and  what 
configurations  may  be  made  available.  The  robotics  program  at  Eort  Hood  was  a  valuable 
resource  for  the  development  of  proposed  systems  and  employment  scenarios  (“Robotics 
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at  Fort  Hood,”  n.d.).  The  question  arises:  how  should  the  Army  examine  these  ideas  in 
order  to  make  eritieal  deeisions  about  how  or  if  the  concept  will  be  developed? 

Three  candidate  scenarios  were  selected  for  Wingman:  chase  (or  reconnaissance), 
attack,  and  defend.  For  each,  we  developed  a  scenario  in  VBS2  as  an  exemplar  of  the 
problem  set  unique  to  each  situation  (see  Table  2).  In  this  study,  player  participants  were 
not  able  to  modify  the  scenarios  or  vehicle  configurations.  We  did,  however,  ask  several 
questions  about  both  of  those  issues  in  the  debrief  interviews. 

The  general  script  that  we  followed  in  conducting  the  study  can  be  seen  in 
Appendix  D,  and  the  layout  of  the  lab  area  where  the  game  sessions  were  conducted  is  in 
Appendix  E.  The  pre-game  demographic  survey  and  post-play  feedback  surveys  can  be 
seen  in  Appendices  F  and  G,  respectively.  Appendices  H  through  M  contain  summary 
transcriptions  of  the  audio  recordings  made  during  the  pilot  and  five  game  sessions. 


Table  2.  Wingman  Virtual  Battlespace  Simulation  Version  2  Scenarios  and 

Descriptions 


Scenario 

Player  Breakout 

Geo-Location 

Additional  info 

Chase 

Four  BLUFOR 

Takistan 

Night 

Narrative: 

A  FIIGFI-RANKING  OPFOR  OFFICER  is  in  the  area  of  Takistan,  exact  whereabouts 
unknown.  A  convoy  in  village  of  Flazar  Bagh  will  meet  him.  Using  APDs,  follow  the 
convoy  without  being  seen.  When  contact  with  the  target  is  made,  eliminate  him, 
all  units  supporting  him,  and  the  convoy. 

Attack 

two  BLUFOR,  two  OPFOR 

Geotypical  Eastern  Europe 

OPFOR  Al 

Narrative: 

BLUFOR  in  the  vicinity  of  a  heavily  guarded  enemy  compound  used  as  a  prison  for 
friendly  forces/individual. 

Defend 

two  BLUFOR,  two  OPFOR 

Porto 

BLUEFOR  and  OPFOR  Al 

Narrative: 

BLUFOR  defending  a  position  in  the  vicinity  of  Porto  against  a  large  but  unknown 
number  of  enemy  fighters. 

The  study  consisted  of  a  pilot  followed  by  five  groups  of  four  participants  each 
approved  by  IRB  protocol  NFS. 2014. 0026-CR01-EP7-A.  The  24  volunteer  participants, 
in  the  pilot  and  subsequent  sessions,  were  experienced  military  officers  at  NFS,  but  not 
all  were  experienced  in  a  ground  combat  discipline.  (See  Table  3  for  player  demographic 
data.)  Computer  game-play  frequency  ranged  from  none  (eight  respondents)  to  over  five 
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hours  per  week  (four  respondents).  Each  session  took  two  to  three  hours  to  complete. 
During  game  play,  players  were  co-located  where  they  could  talk  to  each  other  directly. 
For  the  attack  and  defend  scenarios,  we  moved  the  opposing  enemy  forces  (OPFOR) 
team  to  another  room  so  that  they  could  not  hear  the  other  team  during  planning  or 
scenario  execution. 

Table  3.  Demographic  Data  and  Game  Experience  of  Volunteer  Participants 


Player  Demographics 

Sex 

Male 

Female 

23 

1 

Service 

USA 

USN 

Canadian  Army 

3 

7 

13 

1 

Rank 

0-1 

0-2 

0-3 

0-4 

0-5 

1 

2 

14 

7 

1 

Years  of  Service 

1-5 

6-10 

11-15 

16+ 

2 

11 

9 

2 

Specialty 

19  different  service  specialties 

Game  Experience 

None 

<1  Hr/wk 

1-2  Hr/wk 

5+  Hr/wk 

8 

9 

3 

4 

We  refer  to  Wingman  as  an  autonomous  platform  demonstrator,  or  APD.  The  key 
parameters  available  to  configure  each  APD  are  lethality  (weapons),  vulnerability 
(armor),  and  agility  (engine  and  weight).  Figure  15  depicts  the  APDs  and  the 
environment  for  the  familiarization  scenario  in  VBS2.  We  pre-configured  three  versions 
of  the  APD  for  this  study,  each  with  a  different  balance  of  lethality,  vulnerability,  and 
agility  (see  Table  4). 
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Figure  15.  Autonomous  Platform  Demonstrators  Employed  in  VBS2 


Table  4.  Autonomous  Platform  Demonstrator  Configurations 


Name 

Armor  Level 

Weapon 

Engine  Power  Level 

APD  25mm/7.62 

Moderate 

M240 

Moderate 

APD  Ml 34 

Moderate 

Ml 34  Minigun 

Moderate 

APD  Speedy 

Eow 

M240 

High 

We  used  the  standard  desktop  PC  eonfiguration  of  VBS2  with  mouse  and 
keyboard  inputs.  Eaeh  group  was  first  read  general  instruetions  pertaining  to  the  study 
and  information  about  the  Wingman  eoneept  vehicle.  Each  player  was  provided  a  “cheat” 
card  keyboard  function  map  for  VBS2  with  important  keys  highlighted.  We  provided  a 
familiarization  scenario  that  brought  all  players  to  a  minimum  level  of  competence  at 
using  VBS2  both  in  APD  mode  and  dismounted  mode,  which  are  always  operated 
separately  (players  either  operate  their  APD  or  move  their  soldier,  but  not  both 
simultaneously).  Prior  to  each  scenario,  we  gave  specific  instructions  as  to  objectives  of 
that  scenario  and  answered  player  questions.  We  provided  no  coaching  during  scenarios, 
but  we  did  answer  questions  related  to  the  VBS2  interface.  We  gave  teams  five  minutes 
before  each  scenario  to  develop  a  plan.  We  showed  all  players  on  each  team  the  tactical 
map  of  the  area  in  VBS2,  as  well  as  the  initial  positions  of  each  player  and  his  or  her 
equipment  on  that  map.  We  informed  groups  that  they  could  play  any  scenario  as  many 
times  as  they  collectively  wished. 
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During  game  play,  we  reeorded  all  verbal  eommunication  for  later  analysis.  We 
also  used  the  after-aetion  review  (AAR)  capabilities  in  VBS2.  The  AAR  from  VBS2 
stores  the  game  session  in  real-time  and  can  be  replayed  in  a  two  dimensional  (2D)  map 
view,  a  three  dimensional  (3D)  model  view,  or  any  combination  of  those  options.  For  this 
study,  there  were  a  few  automatic  collection  data  points,  such  as  rounds  fired  and  the 
status  of  both  enemy  and  friendly  killed  and  wounded.  At  the  conclusion  of  play,  we 
brought  all  the  participants  together  for  a  moderated  debrief  session.  Questions  centered 
on  two  main  topics — those  specific  to  the  Wingman  concept  and  those  specific  to  ESP. 

B,  RESULTS 

1.  Chase  Scenario 

The  chase  scenario  was  preceded  by  a  familiarization  scenario.  The  narrative  and 
scenario  objectives  were  provided  followed  by  game  play  and  a  group  after-action  group 
interview  session  to  collect  player  feedback. 

a.  Narrative  and  Objectives 

Recent  Intel  suggests  that  A  HIGH-RANKING  OPFOR  OFFICER  is  currently  in 

the  area  of  Pakistan,  although  his  exact  whereabouts  are  unknown.  A  convoy  in  the  small 
village  of  Hazar  Bagh  is  set  to  meet  him.  Using  your  APDs,  follow  the  convoy  without 
being  seen.  When  contact  with  the  target  is  made,  eliminate  him,  all  units  supporting  him, 
and  the  convoy.  Remain  under  cover  and  avoid  detection.  Eliminate  OPFOR  once  contact 
with  THE  TARGET  is  made.  No  APDs  can  be  destroyed. 

b.  Scenario  Analysis 

This  scenario  was  challenging  in  that  it  had  two  distinct  phases  that  had  different 
requirements  for  the  APD.  During  the  chase  phase,  players  wanted  a  faster  APD  even  if 
they  had  to  give  up  armor  or  lethality.  But  when  they  reached  the  end  of  the  scenario  and 
had  to  successfully  eliminate  the  target  and  his  supporting  units,  players  needed  an  APD 
with  armor  and  lethality.  The  APD  characteristics  players  sought  for  different  missions 
suggested  that  players  might  not  want  APDs  with  identical  configurations. 

Because  stealth  is  a  mandatory  element  of  this  scenario,  players  had  to  master  the 

night  vision  and  lighting  features  in  VBS2.  The  chase  scenario  was  also  the  first  scenario 
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after  the  familiarization  that  every  group  exeeuted.  Consequently,  there  was  more  of  a 
learning  eurve  here  than  in  the  following  two  scenarios. 

A  successful  approach  to  this  scenario  utilized  bounding  overwatch  with  one  APD 
to  the  north  of  the  road,  one  to  the  south,  and  another  trailing  the  convoy  far  enough  back  to 
not  be  detected  but  close  enough  to  be  able  to  support.  As  shown  in  Figure  16a,  the  convoy 
(5)  travelled  along  the  road  between  villages.  It  stopped  at  point  5b,  which  is  where  the 
target  appears.  This  group  had  APD4  to  the  north  of  the  road  and  APD3  to  the  south.  APD4 
kept  to  the  high  ground  in  order  to  observe  all  movement.  APD3  also  tried  to  remain  in 
visual  contact  but  was  less  successful.  APDl  trailed  the  convoy  all  the  way  to  the  end,  and 
APD2  remained  behind  as  rear  guard.  When  the  target  appeared  at  5b,  players  were  alerted 
that  they  should  now  attack  the  convoy.  The  target  and  supporting  units  moved  to  5  c  where 
they  were  engaged  by  all  four  APDs.  The  players  de-conflicted  their  fires  verbally  and 
through  direct  observation.  This  group  had  positioned  its  APDs  to  bind  the  target  on  three 
sides.  During  the  ensuing  firelight,  APDl  was  mostly  offensive  while  APD3  and  APD4 
prevented  the  convoy  from  any  chance  of  egress  from  the  area.  APD2  moved  into  the 
engagement  area  from  its  rear  guard  position  and  was  able  to  support  APDl.  Figure  16b 
shows  the  situation  immediately  prior  to  the  firelight  at  the  end  of  this  run. 

Although  successful,  this  group  jacked  into  its  designated  APDs  immediately  at 
the  start  of  the  scenario  while  the  virtual  soldiers  remained  in  the  starting  locations.  This 
group  maintained  the  perspective  of  robot  controller  throughout  game  play  and  never 
returned  to  the  Soldier  perspective.  The  human  operators  (their  avatars)  were  not  directly 
employed.  Did  they  need  to  be  on-site?  One  of  the  player  groups  provided  feedback, 
suggesting  that  a  local  controller  was  essential,  and  three  explicitly  sought  an  off-site 
controller.  The  other  two  groups  did  not  offer  a  direct  opinion  on  the  local  vs.  off-site 
discussion.  Had  an  APD  been  disabled  in  the  firelight,  an  on-site  Soldier  could  move  into 
position  to  physically  support  achievement  of  the  objective.  An  off-site  Soldier  could  not 
physically  assist  but  may  have  had  a  wider  vantage  point  for  battlefield  situational 
awareness  to  provide  support  from  afar. 
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Figure  16.  Map  (a)  and  After-Action  Review  (b)  View  of  a  Successful  Chase 

Scenario 


A  problem  that  we  saw  in  75%  of  group  runs  was  a  failure  to  remain  undetected. 
This  typically  occurred  when  an  APD  would  follow  too  closely  because  a  player  would 
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underestimate  the  noise  signature  or  aeeeleration  rate  of  the  APD.  At  the  other  extreme, 
after  a  delay  to  allow  for  preparation,  the  OPFOR  eonvoy  moved  out.  If  Blue  (friendly) 
foree  (BLUFOR)  APDs  lagged  too  far  behind,  they  lost  eontaet  with  the  enemy  eonvoy, 
whieh  ended  the  seenario. 

Every  attempt  at  this  seenario  was  charaeterized  by  some  level  of  disorientation  at 
the  onset  of  the  run.  Ten  players  reported  spatial  disorientation  in  free-form  survey 
eomments,  and  all  players  verbally  expressed  ehallenges  during  play.  Even  after  viewing 
a  map  and  knowing  the  loeation  of  both  the  Soldier  and  APD,  the  diseontinuous  switeh 
from  a  geoeentrie  view  of  the  tactical  map  to  the  egocentric  view  in  the  APD  was 
difficult  to  overcome.  Allowing  the  players  multiple  attempts  at  this  scenario  helped.  By 
the  second  or  third  attempt,  most  players  reduced  their  disorientation  and  executed  their 
plan.  In  all,  only  two  of  24  players  were  unable  to  effectively  navigate  in  the  game 
environment  to  meet  the  objectives  designated  in  their  group  plan  for  the  third  attempt. 

The  visual  display  itself  contributed  to  the  disorientation.  The  direction  of 
movement  (shown  in  an  inset  view  by  default)  and  the  direction  of  the  turret  and  optics 
(shown  in  the  main  view  by  default)  are  controlled  separately.  The  variant  view 
perspectives  created  confusion  for  players.  Players  would  think  they  were  lined  up  to 
move  forward  on  the  road,  but  when  they  pressed  the  key  to  move,  they  would  see  that 
the  turret  was  not  aligned  with  their  wheels.  Overcoming  the  disparity  in  view 
perspective  took  practice  and  was  the  subject  of  several  recommendations  made  during 
debrief  (see  Chapter  IV,  Section  C). 

2,  Attack  Scenario 

Both  BEUEOR  and  OPEOR  were  provided  objectives  for  the  Attack  Scenario. 

a.  Narrative  and  Objectives 

BEUEOR;  The  prisoners  will  start  running  to  freedom  once  the  shooting  begins, 
but  the  OPEOR  will  shoot  at  them  as  they  run  away.  Rescue  as  many  prisoners  as 
possible.  Eliminate  all  of  the  OPEORs. 
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OPFOR;  Two  BLUFOR  operatives  with  APDs  are  attacking.  You  must  stop  them 
from  freeing  the  prisoners,  but  do  not  shoot  the  prisoners,  even  if  they  have  escaped. 
Eliminate  both  of  the  BLUFORs. 

b.  Scenario  Analysis 

This  scenario  presented  the  first  opportunity  for  players  to  fight  against  each 
other.  We  immediately  saw  that  having  live  OPFOR  improved  the  level  of  player 
engagement  on  both  sides.  Because  the  ESP  concept  requires  players  to  be  highly 
motivated  to  play,  this  was  an  important  observation. 

There  was  a  much  broader  set  of  strategies  used  to  attempt  to  defeat  the  OPEOR. 
The  BEEIEOR  is  badly  outnumbered  15  to  one  in  this  scenario,  so  a  direct  assault  had 
little  chance  of  success.  To  augment  the  two  live  OPEOR  players,  13  additional  artificial 
intelligence  (AI)  agents  were  included  in  the  scenario.  The  AI  OPEOR  were  armed  with 
shoulder-fired  rocket  propelled  grenades  (RPG)  capable  of  disabling  an  APD  with  a 
single  shot.  Two  common  strategies  involved  either  stand-off  and  sniping  and/or  a 
flanking  maneuver  to  get  behind  the  compound. 

In  Eigure  17a  and  17b,  we  show  one  group’s  attempt  to  flank  the  OPEOR.  APDl 
moved  a  short  distance  to  high  ground  with  cover  and,  after  a  delay  to  allow  APD2  to  get 
into  position,  engaged  targets  from  a  distance.  APD2  took  a  circuitous  route  around 
APDl  to  the  north  ending  on  high  ground  behind  the  compound.  APDl  then  proceeded 
south  along  the  back  fence,  remaining  concealed  whenever  possible.  APD2  purposely 
held  fire  until  the  last  possible  moment.  The  two  OPEOR  players  had  no  idea  where  the 
APDl  was  navigating  until  the  APD  appeared  behind  them. 

Once  shots  are  fired,  the  prisoners  in  the  compound  attempt  to  flee.  If  enough 
guards  have  been  eliminated,  BEUEOR  achieves  their  mission  objective.  If  not,  the 
guards  will  shoot  the  prisoners  rather  than  allow  them  to  escape.  In  this  instance,  APDl 
and  APD2  had  cleared  enough  guards  to  allow  the  prisoners  to  flee.  The  prisoners  exit  the 
compound  to  the  east  heading  down  the  road  that  runs  just  south  of  APDl ’s  position. 
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BLUF0R2  St*rt 


Figure  17.  Map  (a)  and  After- Action  Review  (b)  View  of  a  Successful  Attack 

Scenario 

The  attack  scenario  was  less  disorienting  than  the  chase,  probably  due  to  daylight 
conditions.  We  observed  far  less  wasted  time  at  the  beginning  of  each  run  in  contrast  to 
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the  erratic  movement  patterns  and  volatile  changing  of  view  perspectives  previously 
observed.  Groups  had  a  plan,  and  they  executed  it.  The  OPFOR  players  were  also  readily 
able  to  develop  and  execute  a  plan.  One  OPFOR  group  left  defense  of  the  compound  to 
the  AI  OPFOR  and  went  out  to  hunt  the  APDs  and  their  operators.  This  strategy  was  not 
very  successful  because  once  the  players  fired  on  an  APD  from  outside  the  compound, 
they  were  detected.  The  BLUFOR  quickly  eliminated  their  opponents  with  APDs  by 
quickly  employing  effective  long  range  sites  and  superior  weaponry.  Map  usage  when 
moving  the  avatar  or  APD  was  particularly  important  in  this  scenario. 

Another  difference  in  this  scenario  was  that  each  BLUFOR  player  was  given  a 
truck  to  quickly  move  his  or  her  Soldier  to  a  new  location  if  desired.  Unlike  the  Chase 
scenario  where  it  was  unlikely  that  BLUFOR  would  lose  an  APD  until  the  firelight  at  the 
very  end,  in  Attack,  a  BLUFOR  player  could  lose  his  or  her  APD  at  any  time.  Therefore, 
the  positioning  of  the  Soldier  became  more  important  to  ensure  cover  and  concealment 
while  maintaining  situational  awareness  of  the  battlefield.  Even  if  the  APD  was  lost,  the 
player  still  had  an  armed  Soldier  to  complete  the  mission. 

The  APD  offered  a  large  visible  target  for  the  OPFOR’s  rocket-propelled 
grenades.  As  such,  more  than  half  of  the  players  suggested  that  the  APD  might  be  an 
excellent  diversion  to  cover  the  actual  movement  of  forces.  If  a  group  of  APDs  was 
inexpensive  and  essentially  disposable,  it  might  be  capable  of  masking  the  movement  of  a 
large  force.  Another  suggestion  was  to  protect  the  APD  with  armor  but  not  arm  it  with  a 
weapon  at  all.  Instead,  it  could  be  used  as  a  mobile  shield  and  supply  transport  for  foot 
mobile  soldiers  during  movement  to  contact. 

3.  Defend  Scenario 

The  Defend  scenario  was  the  last  scenario  and  the  least  preferred,  as  informally 
assessed  during  post-play  comments  and  feedback. 

a.  Narrative  and  Objectives 

Defend  your  position  from  OPFOR  units  for  five  minutes.  Each  of  you  have  an 
APD  under  your  control.  If  the  APD  is  destroyed,  you  will  lose  5  points  and  the  APD  will 
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automatically  re-spawn  in  its  original  positions.  If  either  player  dies,  you  will  lose  10 
points  and  the  enemies  will  all  be  reset  to  their  original  positions.  Rescue  as  many 
prisoners  as  possible. 

b.  Scenario  Analysis 

In  the  Defend  scenario,  APDs  were  used  to  protect  an  unfortified  position  from  an 
attack  of  a  numerically  superior  force.  BLUFOR  typically  positioned  its  APDs  in  front  of 
the  position  with  avatars  protected  behind  a  solid  wall.  In  this  scenario,  multiple  APDs 
were  made  available  so  that  when  one  was  lost,  a  replacement  was  immediately  available 
for  use. 

This  scenario  offered  the  least  variation  in  strategy  due  to  the  parameters  of  the 
scenario  design.  The  large  OPFOR  with  RPGs,  along  with  the  availability  of  multiple 
APDs,  often  resulted  in  an  APD  “graveyard”  in  front  of  the  unfortified  position.  In  all 
sessions,  the  default  strategy  was  to  rely  on  the  superior  armor  and  firepower  of  the  APDs 
to  both  attack  the  oncoming  OPFOR  and  defend  the  position.  Movement  away  from  the 
position  would  have  left  it  far  too  vulnerable  given  the  number  of  OPFOR  and  the  speed 
with  whieh  they  attacked  the  position. 

Players  were  least  likely  to  request  to  play  this  scenario  again.  Based  on  its 
position  as  the  final  scenario,  player  fatigue  may  have  played  a  role  in  the  aversion  to 
continuing  with  the  defend  scenario.  An  additional  feature  that  turned  players  off  from 
this  seenario  was  a  re-spawn  feature  factored  into  the  game.  When  the  OPFOR  players 
were  re-spawned,  they  did  not  always  go  back  to  their  original  eharacter,  leading  to 
disorientation  and  eonfusion  on  how  to  best  execute  or  continue  exeeution  of  the 
scenario.  The  BLUFOR  players  and  their  APDs  were  re-spawned  as  well.  Despite 
returning  to  the  same  player  within  close  proximity  to  their  previous  location,  the  re¬ 
spawn  resulted  in  considerable  disorientation  and  dissatisfaction  from  players. 

C.  DISCUSSION 

The  recommendations  received  from  player  teams  were  foeused  on  (a)  the 
Wingman  concept  and  (b)  ESP  and  the  utility  of  game  environments  to  explore  new 
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concepts.  The  program  offiee  defined  the  eritieal  output  as  the  eoneept  of  operations 
(CONOPs)  of  how  Soldiers  most  effeetively  used  the  APD  and  the  eharaeteristics  of 
Wingman  that  allowed  them  to  be  suecessful.  Of  secondary  importance  was  the  ability  to 
vary  vehiele  and  scenario  parameters  to  observe  and  measure  the  value  of  different 
performanee  metrics. 

1.  Wingman  Concept 

Although  our  scenarios  in  this  study  focused  on  only  three  pre-eoneeived  methods 
of  employment  for  Wingman,  our  players  expanded  that  list  signifieantly  during  the 
debriefing: 

•  Reconnaissance:  APD  has  a  greater  ability  to  be  aggressively  applied 
without  fear  of  easualty  or  eapture.  It  eould  be  eonfigured  with  multiple 
sensors,  video  eommunieations,  or  armament.  Ground-based 
reeonnaissanee  may  also  be  a  powerful  alternative  to  UAV  reeonnaissanee 
when  weather  interferes  with  the  mission. 

•  Transport:  Over  short  distanees,  APD  is  useful  for  transporting  people, 
equipment,  ammunition,  and  supplies. 

•  Ambush:  APD  is  maneuvered  to  a  strategic  position  and  could  lie  in  wait 
indefinitely. 

•  Breaching:  APD  eould  be  eonfigured  speeifieally  for  breaehing  walls, 
fortified  doorways,  or  other  strong  points  that  are  typieally  dangerous  for 
personnel  to  breaeh. 

•  Mobile  Mine:  If  light  and  inexpensive,  APD  eould  be  armed  with 
explosives,  quietly  maneuvered  into  the  adversary’s  position,  and 
detonated. 

•  Defense:  As  an  unmanned  patrol,  APDs  eould  be  used  to  seeure  or  expand 
the  perimeter.  Players  pointed  out  that,  unlike  with  a  manned  perimeter, 
one  would  want  the  APD  to  take  fire  beeause  this  identifies  the  position 
and  possible  strength  of  the  adversary. 

•  Attack:  APD  must  be  reeonfigurable,  possibly  even  within  a  mission  (e.g., 
switeh  from  one  weapon  to  another). 
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ESP  appears  to  be  an  exeellent  way  to  explore  human  interfaees  with  systems.  As 
eonfigured  in  this  study,  Wingman  has  limitations,  but  we  were  eneouraged  that  the  game 
environment  was  so  effeetive  at  identifying  what  Soldiers  would  want  the  interfaee  to  be 
able  to  do.  Among  the  recommendations  were 

•  Navigation:  Simplify  the  controller  by  allowing  waypoints  to  be  used  so 
that  the  APD  could  auto-drive  along  a  preset  path  to  an  objective 
bypassing  obstacles  through  the  use  of  sensors.  The  APD  could  have  an 
autonomous  mode  that  would  allow  the  operator  and  APD  to  function 
simultaneously.  A  map  chip  would  be  useful. 

•  Controller:  Wingman  needs  a  specialized  controller.  The  complexity  of 
the  mission  did  not  map  well  to  a  conventional  keyboard  and  mouse.  Most 
players  felt  that  a  typical  game  controller  would  be  more  appropriate. 

•  “Pre-sets”:  Wingman  is  a  complex  vehicle.  There  are  (or  could  be) 
multiple  configurations  for  sensors,  navigation,  or  weapons.  Players 
wanted  to  be  able  to  pre-set  several  configurations  and  then  switch  to  each 
with  a  single  button  rather  than  work  through  menus  to  turn  lights  off, 
activate  night  vision  goggles  (NVG),  and  so  forth. 

There  was  a  lively  discussion  about  whether  the  Wingman  APD  should  be  a 
ground-based  unmanned  vehicle  where  the  operator  remains  at  a  safe  distance  from 
danger  using  a  sophisticated  interface  to  control  the  vehicle  as  opposed  to  being 
controlled  by  an  on-site  Soldier  who  could  directly  see  the  APD.  A  related  issue  was  the 
complexity  of  managing  the  turret  and  the  navigation  concurrently.  Some  players 
suggested  that  maybe  Wingman  should  have  two  operators — one  for  the  turret/weapon 
and  one  for  the  navigation.  Players  voiced  concern  about  how  Wingman  alerts  the 
operator  of  its  damage  status  while  employed  and  then  how  and  at  what  level 
maintenance  could  be  accomplished.  As  the  APD  takes  fire,  the  interface  does  not  present 
system  “health”  status  to  the  player  (e.g.,  status  of  mobility,  sensors,  communications, 
fuel/battery,  ammo,  and  weapons).  Lastly,  players  commented  on  the  potential  of  creating 
a  moral  hazard  concerning  the  use  of  a  lethal  robotic  weapon  system  against  a  human 
adversary. 
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2,  Early  Synthetic  Prototyping  Concept 

Players  almost  universally  endorsed  the  concept  of  using  game  environments  as  a 
testing  ground  for  early  ideas,  with  23  of  24  reporting  that  they  would  play  this  or  similar 
games  in  the  future.  However,  there  were  artificialities  in  our  study  that  ESP  will  need  to 
address,  such  as  the  colocation  of  players.  They  spoke  freely  to  each  other  before,  during, 
and  after  each  scenario.  ESP  could  use  voice  over  internet  protocol  (VOIP) 
communications  to  enable  speaking,  but  capture  for  analysis  would  be  limited  to 
processing  of  automated  transcriptions  without  the  ability  to  observe  and  capture 
inflection.  Also,  we  gathered  our  players  for  a  detailed  AAR  debriefing  session.  ESP  is 
distributed;  therefore,  if  there  is  an  AAR  mode,  it  must  be  meaningful  to  players  or  they 
will  simply  skip  it.  Our  VBS2  implementation  was  instrumented  only  as  far  as  the  built- 
in  AAR  was  instrumented,  limited  to  shots  fired  and  virtual  enemy  killed  in  action  (KIA) 
or  wounded  in  action  (WIA).  It  is  imperative  that  ESP  allow  the  scenario  author  to 
specify  metrics  to  be  captured. 

We  asked  players  whether  they  would  have  preferred  to  configure  their  APDs 
from  components  rather  than  be  given  a  specific  APD  for  each  scenario.  Most  said  they 
would  have  preferred  to  build  their  own,  especially  if  they  had  attempted  scenarios 
multiple  times.  After  one  run,  a  player  starts  to  understand  the  limitations  of  a 
configuration  and  is  able  to  express  what  additional  characteristics  are  desirable.  Play 
also  provided  insights  into  additional  information  that  the  players  would  like  to  have 
known  about  the  status  of  their  APDs.  Eor  example,  after  riding  over  rough  terrain,  a 
player  was  interested  in  the  level  of  damage  the  vehicle  had  sustained  from  the  rocks  and 
foliage  the  vehicle  had  crossed  on  its  path.  Another  proposal  was  to  incorporate  waypoint 
movement  for  the  vehicle  to  reduce  the  tactical  navigation  burden  on  the  operator  by 
incorporating  use  of  GPS  coordinates  to  lay  down  an  efficient  travel  route. 

The  ESP  concept  encourages  players  to  explore  new  equipment  and  scenarios. 
Again,  we  were  encouraged  by  player  responses,  which  ranged  from  recommendations  to 
improve  the  resilience  of  the  armor  to  the  proposed  employment  scenario  as  a  resource  in 
support  of  village  stability  operations.  We  asked  players  whether  they  would  author  new 

scenarios  or  modify  existing  scenarios,  provided  they  were  given  the  tools  to  do  so.  There 
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was  a  mixed  response  to  this.  Some  players  naturally  want  to  design  new  “levels,”  others 
just  want  to  play  and  master  the  levels  provided.  We  were  eneouraged  to  hear  that  a  large 
group  of  players  viewed  seenario  editing  as  a  part  of  the  game,  or  as  a  reason  why  they 
would  want  to  play  ESP  games;  it  is  a  part  of  the  eompetition. 

We  asked  about  the  fidelity  of  the  game  environment.  Players  eommented  on 
artifaets  of  VBS2  that  were  somewhat  irritating.  For  example,  an  APD  beeame  stuek  on  a 
small  bush  beeause  VBS2  represented  it  as  a  solid  obstaele  when  the  real  APD  would 
have  easily  rolled  over  it.  Some  aspects  of  the  lack  of  fidelity  point  to  needed 
improvements  to  retain  player  buy-in.  For  example,  in  the  attack  scenario,  one  player 
attempted  to  achieve  cover  and  concealment  within  a  building  in  the  scenario;  the  fidelity 
of  the  VBS2  environment  we  used  did  not  accommodate  this  tactically  sound  decision. 
Players  being  stymied  by  the  lack  of  fidelity  occurred  again  in  the  defend  scenario  when 
players  placed  their  avatars  on  the  roof  of  a  building  to  gain  a  better  vantage  point  from 
which  to  observe  enemy  movement.  When  the  player  attempted  to  take  cover,  he  was 
unable  to  control  his  APD.  Finally,  there  were  consistent  negative  remarks  about  trying  to 
control  a  combat  system  using  a  desktop  computer  and  keyboard  commands  rather  than  a 
game  controller  or  joystick.  Despite  these  readily  apparent  examples  of  the  limitations  of 
fidelity  in  the  study,  players  readily  adapted  to  these  limitations  and  appeared  to  be 
reasonable  in  their  expectations  and  in  drawing  conclusions  about  the  system  being 
considered  based  on  what  they  saw  in  the  scenarios.  See  Table  5  for  results  from  several 
selected  survey  questions.  The  survey  requested  responses  on  the  interval  of  1  to  5;  1 
indicating  yes  with  certainty,  and  5  indicated  no  with  certainty.  Questions  with  response 
rates  recorded  as  a  percentage  were  based  on  respondents  providing  a  strictly  yes  or  no 
response. 
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Table  5.  Response  Data  from  Selected  Survey  Questions. 


Survey  Questions  (selected) 

Avg  response 
(scale  1-5) 

1  Do  you  feel  that  your  time  was  spent  effectively  today? 

1.35 

Would  you  contribute  to  future  efforts  to  develop  or  test  ideas  in  a  game 
2  environment? 

1.7 

Would  you  encourage  others  to  participate  in  future  efforts  to  develop  or 
3  test  ideas  in  a  game  environment? 

1.5 

4  Would  you  play  this  game  or  one  very  similar  to  it  again? 

95% 

How  often  would  you  play  if  you  had  access  ... 

4a  ...  via  download  to  personal  device? 

65% 

4b  ...  via  internet? 

65% 

4c  ...  at  work  as  well  as  designated  time  to  play?  * 

80% 

*  75%  of  respondents  in  this  category  would  play  more  than  once  a  week 

3,  Study  Conclusions 

Our  ESP  study  produced  valuable  contributions  to  an  actual  acquisition  program. 
Proposed  tactics,  techniques,  and  procedures  for  system  employment  were  included  in 
these  contributions,  demonstrating  that  ESP  can  excel  in  the  areas  of  concern  to  the 
program  office,  evaluation  of  CONOP,  and  helping  to  build  context  for  the  proposed 
mission  of  the  equipment.  ESP  provides  a  means  to  link  capabilities  to  outcomes.  We 
created  enough  realism  to  elicit  novel  ideas  that  were  testable  and  would  readily  scale  up 
to  a  distributed  game  environment.  Players  clearly  wanted  to  play,  they  wanted  to  win, 
and  they  were  more  than  willing  to  talk  about  the  experience.  We  found  that  individual 
motivation  and  friendly  rivalry  sparked  incentive  to  meet  the  mission  objectives  and 
pursue  successful  employment  strategies.  Subsequent  player  comments  and  feedback 
were  triggered  by  their  play  experience.  Accepting  the  limitations  of  a  game  environment 
is  critical.  ESP  scenarios  are  admittedly  crude,  yet  we  gained  important  operator  feedback 
at  an  early  phase  of  development  at  extremely  low  cost. 
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V.  RECOMMENDATIONS  AND  CONCLUSION 


The  study  we  conducted  for  ESP  is  not  without  its  limitations.  ESP  itself  is 
currently  at  the  conceptual  stage,  and  in  our  attempt  to  determine  how  it  should  function 
and  what  expectations  we  might  reasonably  have  of  the  eventual  system,  we  used 
surrogates  that  are  similar  to  but  not  exactly  what  ESP  is  envisioned  to  be.  We  must 
address  any  artificialities  and  limitations  imposed  by  those  surrogates,  be  they  systems  or 
processes.  Those  limitations  invite  further  study  to  explore  the  possibilities  for  collecting 
player  feedback  and  input  toward  bettering  a  future  capability.  Despite  limitations,  we 
were  able  to  compile  several  cogent  recommendations  for  the  development  of  the  ESP 
framework  as  the  Army  continues  to  pursue  ESP  as  a  process  and  set  of  tools. 

A,  STUDY  LIMITATIONS 

The  structure  of  the  study  itself  had  limitations  that  affected  the  types  and  quality 
of  data  we  were  able  to  collect.  There  was  no  control  group  to  compare  what  kind  of 
feedback  could  be  elicited  from  a  traditional  focus  group  discussion  for  Wingman  in 
contrast  to  the  feedback  that  was  accumulated  during  and  after  playing  the  scenarios. 
Based  on  the  time  available  for  the  game  trials  at  NPS,  if  a  focus  group  comparison  is 
conducted  in  the  future,  the  time  for  game  play  will  need  to  be  reduced  so  as  not  to  deter 
participants  by  imposing  a  significant  time  commitment.  Incorporating  the  focus  group 
would  have  to  be  a  between  group  design,  not  a  within  group  design  because  of  the 
learning  effects  inherent  in  the  study. 

Eurthermore,  the  nature  of  the  study  we  conducted  did  not  allow  for  us  to  draw 
conclusions  about  how  play  would  be  accomplished  and  data  collected  in  a  truly 
distributed  game  environment.  To  assess  a  semblance  of  true  distribution  in  a  lab 
environment,  player  communication  would  need  to  be  restricted  to  VOIP  using 
headphones  and  speakers  without  the  ability  to  observe  the  screen,  verbal  observations,  or 
physical  gestures  of  peer  players.  Actual  physical  dispersion  does  not  allow  for  physical 
assessment  of  other  players,  either  peer  or  opponent.  Our  players  did  not  know  each  other 
personally  in  most  cases,  but  they  were  aware  of  the  others’  status  as  a  military  officer 
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and  may  have  made  inferenees  about  the  other  players  eompetenee  based  on  that 
information.  We  eannot  eompletely  diseount  the  effeet  of  the  military  rank  of  our  players, 
but  we  noted  very  open  diseussions  with  lower  ranking  players  freely  eontradieting 
higher  ranking  officers  (respectfully)  without  hesitation.  We  felt  the  effect  of  rank  was 
minimal,  if  present  at  all.  Beyond  sizing  up  the  competition,  being  able  to  rapidly  address 
and  troubleshoot  connectivity,  sound,  or  other  technical  challenges  in  a  close 
environment  in  the  lab  will  not  emulate  the  myriad  of  connectivity  issues  that  could  arise 
for  actual  players  on  a  distributed  network. 

The  data  collection  methods  we  employed  for  the  NFS  study  are  insufficient  to 
mimic  data  collection  possibilities  in  a  distributed  game  environment.  We  collected  voice 
recordings  and  VBS2  after-action  play  data  and  play-back.  Recorded  voice  data  is 
cumbersome  to  analyze  manually.  Analysis  of  voice  data  in  a  distributed  game 
environment  is  possible  but  faces  the  challenge  of  players  being  reluctant  to  agree  to 
being  recorded.  Any  data  collected  would  require  parsing  recordings  for  key  words,  and 
the  results  obtained  for  analysis  may  fall  short  of  the  effort  required  to  identify  potentially 
relevant  information.  A  better  method  of  data  collection  would  mirror  the  techniques 
used  by  Microsoft  with  their  data  collection  from  specific  data  fields  instrumented  into 
Xbox  games  and  streamed  back  from  on-line  players.  A  joint  effort  between  S&T 
researchers  and  the  assigned  program  office  would  be  necessary  to  determine  the 
elements  critical  for  collection  to  avoid  archiving  volumes  of  unnecessary  data  could  not 
realistically  be  analyzed.  A  follow-on  study  could  set  critical  information  triggers  in  the 
scenarios  for  further  evaluation.  For  example,  in  the  scenarios  we  used  for  Wingman, 
APD  navigation  would  be  a  useful  data  point  and  it  could  be  assessed  by  capturing  route 
data  characteristics  such  as  time  and  distance  on-road  or  over  terrain.  Phase  lines  could 
be  set  to  trigger  an  indicator  of  when  they  are  crossed  during  game  play  indicating  a 
player’s  eagerness  to  approach  the  OPFOR  position  or  willingness  to  cede  space  or  time 
for  some  other  advantage.  Other  measures  that  could  be  implemented  are  use  of  avatar, 
use  of  sites  and  view  perspectives,  choice  of  weapon,  and  position,  time,  and  frequency 
of  weapons  engagement. 
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Another  data  collection  method  we  employed  was  the  use  of  a  post-play  group 
interview  and  survey.  Surveys  could  be  automated  and  put  on-line  but  might  not  see  high 
participation  levels  if  completion  is  optional  for  players.  The  in-person  interview  would 
not  be  readily  available  in  a  distributed  environment.  However,  the  proposed  method  for 
ESP  to  bridge  this  gap  is  to  open  a  forum  for  player  interaction  where  players  could  post 
results  and  feedback.  This  forum  or  forums  would  be  an  ideal  venue  for  S&T 
representatives  to  review  and  incorporate  recommended  system  or  scenario  modification 
for  future  play  opportunities.  We  designed  this  study  under  the  assumption  that  the 
quality  and  quantity  of  data  collected  would  be  directly  aligned  with  the  ease  of  which  it 
was  given.  Speaking  is  easier  than  writing.  Players  are  more  willing  to  voice  a  concern 
than  they  are  to  write  it,  especially  if  it  involves  a  detailed  description.  Furthermore, 
many  of  the  suggestions  we  received  involved  a  “back  and  forth”  between  the  interviewer 
and  the  player  to  extract  the  idea  mainly  because  it  may  have  been  unclear  in  the  mind  of 
the  player  at  the  time.  Through  a  question-and-answer  format,  the  idea  solidified  and  was 
able  to  be  presented  for  further  discussion.  The  active  maturing  of  an  idea  through 
question-and-answer  sessions  occurred  several  times  throughout  the  study. 

Limitations  aside,  our  study  contributed  to  the  conclusion  that  relevant 
information  can  be  collected  from  game  play.  Although  some  of  the  data  we  collected 
was  in  an  artificial  form  due  to  the  limits  of  the  study,  we  know  that  useful  data  is  there. 
The  challenge  now  is  to  find  ways  to  extract  it  in  a  distributed  and  passive  format  that  can 
feed  powerful  game  analytic  processes  resulting  in  useful  information  never  available 
prior  to  the  ESP  concept.  That  information  will  be  useful  for  decision-makers  tasked  with 
developing  systems  that  warfighters  will  employ  in  future  military  operations. 
Furthermore,  we  were  also  able  to  open  up  a  line  of  communication  to  a  group  of 
warfighters  and,  from  that  exchange,  develop  recommendations  for  the  successful 
development  of  ESP  as  a  process  and  set  of  tools  for  the  DOD. 

B,  RECOMMENDATIONS 

ESP  has  a  place  in  the  future  of  DOD  acquisition.  ESP  is  a  means  to  collect 
valuable  input  and  feedback  from  warfighters  and  deliver  it  to  decision-makers  within 
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DOD  programs  early  in  the  aequisition  proeess  and  with  relevant  mission  context.  The 
contributions  of  warfighters  can  positively  impact  development  of  future  systems. 

Despite  a  lack  of  overwhelming  quantitative  data  to  support  prototyping  as  a 
means  to  reduce  program  cost  and  schedule  creep,  the  ability  to  consider  requirements 
and  alternatives  provides  a  program  manager  a  useful  tool  that  supports  mitigation  of  risk 
throughout  the  acquisition  process.  A  virtual  environment  offers  one  way  to  make 
prototyping  practical  and  cost-effective  early  in  the  life  cycle  of  any  system.  Before  the 
fidelity  of  refined  physical  attributes  and  precise  measurements  needs  to  be  established,  a 
game  environment  offers  an  opportunity  to  reach  masses  of  potential  contributors  and 
also  to  collect  and  quantify  their  inputs  in  a  feedback  loop  with  S&T  and  the  program 
office. 

The  game  environment  must  be  widely  available  and  must  be  accessible  without 
substantial  technological  leaps  by  the  user.  Ideally,  the  ability  to  launch  directly  from  a 
web  browser  would  allow  the  easiest  access  for  the  broadest  population  of  users. 
Although  the  application  should  be  easy  to  launch,  access  should  be  limited  to  DOD 
users.  Having  a  common  access  card  (CAC)  enabled  site  or  the  ability  to  initially  gain 
access  using  a  CAC  and  then  subsequent  access  could  be  made  using  a  username  and 
password/code/phrase.  As  established  programs  are  fielded,  perhaps  the  DOD-only 
restriction  could  be  lifted  to  allow  for  broader  access  beyond  the  DOD  to  look  at  novel 
use  cases  for  or  modifications  to  existing  equipment  that  might  apply  within  or  outside 
the  military  domain. 

Although  the  objective  of  ESP  and  its  iterations  for  acquisition  programs  is  not 
entertainment,  the  quality  of  game  must  be  somewhat  comparable  to  commercial  games. 
One  of  the  benefits  of  ESP  is  the  ability  to  bring  in  mass  quantities  of  data,  and  without  a 
superior  interface  and  virtual  environment,  potential  users  will  spend  their  time  doing 
other  things  outside  of  running  proposed  equipment  through  prescribed  scenarios.  Much 
like  other  modern,  popular  games,  the  environment  needs  to  be  networked  to  other 
players  in  real  time  and  return  a  data  feed  for  further  evaluation. 
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The  ability  to  network  also  lends  to  the  instrumentation  of  the  game  environment 
for  data  collection  and  the  return  of  that  data  to  S&T  representatives  for  analysis. 
Instrumentation  of  the  games  will  be  critical  so  as  not  to  overwhelm  researchers  with  data 
that  might  lend  to  superior  commercialization  of  the  game  environment  but  does  not 
contribute  significantly  to  the  development  of  the  technology  solution  or  weapon’s 
system  being  evaluated.  A  discussion  forum  would  be  a  valuable  collection  site  to  gather 
feedback  for  both  the  game  and  the  system  under  evaluation.  A  discussion  forum  will 
also  facilitate  collaboration  between  players  and  observation  of  the  tactics  and  techniques 
that  might  arise  from  such  collaboration. 

A  key  concept  that  we  were  not  able  to  explore  sufficiently  in  our  study  was  the 
possibility  of  modifying  the  scenarios  and/or  the  systems  being  evaluated.  Having  the 
ability  to  modify  the  proposed  technology  or  equipment  is  essential  to  accumulating 
relevant  feedback  from  the  crowd  of  warfighters  contributed  to  the  effort  within  the  ESP 
framework.  Being  able  to  change  the  scenario  itself  also  offers  an  important  opportunity 
to  observe  variant  tactics  and  techniques,  but  must  be  done  judiciously  so  as  to  ensure  the 
existing  data  collected  remains  compatible  with  successive  data  collection  efforts  when 
necessary. 

ESP  should  be  considered  an  enabling  capability  and  not  a  mandatory 
requirement  for  fulfillment  by  a  major  defense  acquisition  program  (MDAP)  to  proceed 
to  subsequent  acquisition  milestones.  Ideally,  ESP  can  be  inserted  pre-Milestone  A  to 
support  a  holistic  ERS  effort  to  manage  risk  using  modeling,  simulation,  and  tradespace 
analysis  with  the  benefit  of  gaining  mission  context  for  the  proposed  system. 

C.  CONCLUSIONS 

The  warfighters  of  the  future  will  require  more,  not  less,  technological  innovation 
to  keep  pace  with  allies  and  maintain  a  competitive  advantage  against  potential 
adversaries.  USD(AT&E)  Frank  Kendall  recently  stated,  “If  the  U.S.  military  wants  to 
stay  No.  1,  then  it  can’t  rely  on  the  private  sector  to  fund  cutting-edge,  military-specific 
R&D.  No  matter  how  much  you  reform,”  he  said,  “such  weapons  programs  will  still  take 
many  years  and  taxpayer  dollars”  (Freedberg,  2014,  para.  4).  Engineering  innovative  and 
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resilient  systems  for  the  future  will  require  new  teehnique  to  maximize  effectiveness. 
ESP  can  aid  in  the  evaluation  of  CONOPs  and  help  to  build  context  for  the  proposed 
mission  of  the  system  in  development.  ESP  offers  an  opportunity  to  explore  the  realm  of 
the  possible,  linking  capabilities  to  outcomes,  in  the  inexpensive  tradespace  of  a  virtual 
environment  linking  proposed  capabilities  to  possible  outcomes.  Most  importantly,  the 
ability  to  seek  and  incorporate  the  input  of  the  large,  diverse,  and  experienced  body  of 
warfighters  offers  an  opportunity  to  collect  a  trove  of  data  unsurpassed  by  current 
techniques. 

Gaining  access  to  warfighters  and  their  potentially  groundbreaking  contributions 
is  not  an  insignificant  task.  Games  are  a  serious  option  to  consider  for  the  accumulation 
of  relevant  data  that  will  be  useful  to  decision-makers  developing  future  DOD  systems. 

ESP  is  not  a  singular  solution  with  ambitions  to  rectify  the  inefficiencies  of  the 
DOD  Decision  Support  Eramework.  Rather,  it  will  be  a  valuable  component  to 
maintaining  a  technological  advantage  as  the  DOD  pursues  innovative  solutions  to  meet 
warfighter  need  into  the  future. 

D.  FUTURE  WORK 

A  plan  for  a  future  study  at  NPS  is  being  considered  to  conduct  a  comparison  of 
information  and  feedback  that  can  be  gained  from  a  directed  focus  group  as  compared  to  the 
feedback  that  can  be  collected  from  game  play.  Additionally,  incorporating  simulated 
dispersion  will  be  accomplished  to  gamer  feedback  from  service  member  players  on  their 
perceptions  of  playing  with  new  technologies  in  a  distributed  environment  without  having 
their  in-game  peers  sitting  beside  them.  The  simulated  dispersion  will  be  accomplished  using 
headsets  to  give  players  the  feel  of  how  play  would  be  conducted  off  duty  with  other  players 
from  around  the  country.  The  study  would  continue  to  evaluate  how  a  player  might  actually 
operate  with  the  system  being  developed  and  assessed  within  the  scenario  to  see  what  kind  of 
information  could  be  gathered  from  additional  game  play. 

Collecting  data  useful  for  analysis  will  be  nontrivial  as  ESP  moves  from  a 
controlled  laboratory  to  a  distributed  setting.  The  Army  is  conducting  follow-on  testing 
on  a  larger  scale  with  a  pilot  planned  for  November  2014  where  30  soldiers  from  the 
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Brigade  Modernization  Command  at  Ft.  Bliss,  TX,  will  be  playing  seenarios  to  evaluate 
proposed  future  teehnologies.  With  a  more  robust  and  distributed  system  in  development, 
the  pilot  test  and  a  subsequent  larger  test  with  100  soldiers  playing  will  be  condueted 
using  VBS3  at  the  game  platform.  The  teehnologies  to  be  assessed  are  taetieal  airdrop 
and  a  eoded  spot  laser.  The  conduct  of  the  pilot  and  larger  study  will  follow  the  general 
framework  used  in  our  study  but  will  be  conducted  with  groups  of  enlisted  Army  soldiers 
at  a  base  simulation  center.  The  continuing  study  will  explore  what  kind  of  information 
can  be  obtained  and  whether  it  will  be  useful  for  the  proposed  systems  being  evaluated  in 
the  game  environment. 

The  next  steps  will  be  to  increase  how  players  customize  the  system  being 
developed  and  assessed  in  ESP  and  to  move  to  a  distributed  framework  with 
instrumented  scenarios  to  gather  and  analyze  game  data.  Ultimately,  the  ESP  concept  will 
enable  the  DOD  to  put  warfighters  at  the  center  of  capability  and  concept  development  to 
continue  the  tradition  of  technological  dominance  over  adversaries  into  the  future. 
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APPENDIX  A.  FY2015  BUDGET  DATA  AND  HISTORICAL  DATA 


The  following  figures  and  text  are  from  the  2014  Annual  Report  on  the 
Performance  of  the  Defense  Acquisition  System  (USD[AT&L],  2014)).  Figure  18  was 
also  displayed  in  Chapter  II  and  shows  how  the  president’s  2015  budget  request  was 
broken  out  by  fund  category.  Figure  19  shows  an  increase  in  military  budgets,  including 
RDT&E  in  the  first  decade  of  the  21st  century  followed  by  a  post-war  decline  in  funding. 
RDT&E  is  detailed  in  Eigure  20. 


by  Budget 
Account 


MilCon 
$5.4  . 


Other 

$2.4 


by  Military 
Department 


NOTE:  Budget  amounts  are  In  billions  of  then-year  (unadjusted)  dollars.  OCO  is  not  Included.  "MilCon''  is  Military 
Construction. 


Eigure  18.  Defense  Budget  Breakouts  in  2015  President’s  Budget  Request 

(from  USD[AT&E],  2014,  p.  2) 
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NOTE:  OCO  Is  Included  In  fiscal  year  budgets  before  FY2014  but  not  In  the  current  fiscal  year  (2014)  or  in  the 
FY2015  President's  Budget  figures  (FY  2015-2019).  Budget  amounts  are  adjusted  for  Inflation  and  reported  in 
billions  of  calendar  year  2015  dollars  (CY15$B). 

Figure  19.  Defense  Budget  Accounts:  Historical  and  FBI 5  (FY  1962- 

FY2019)  (from  USD[AT&L],  p.  2) 
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NOTE:  OCO  Is  included  in  fiscal  year  budgets  before  FY2014  but  not  in  the  current  fiscal  year 
(2014)  or  in  the  FY2015  President’s  Budget  figures  (FY  201S-2019).  Budget  amounts  are 
adjusted  for  inflation  and  reported  in  billions  of  calendar  year  2015  dollars  (CY15$B). 

Figure  20.  Recent  and  Projected  RDT&E  Budgets  as  of  PB2015  (FY2000- 

FY2019)  (From  USD(AT&L),  p.  3) 
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APPENDIX  B.  ENGINEERING  RESILIENT  SYSTEMS 
AND  THE  DEFENSE  ACQUISITION  SYSTEM 
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APPENDIX  C.  EARLY  SYNTHETIC  PROTOTYPING 


Acquisition  professionals  (1)  use  scenario  editing  tools  to  develop  eoncept  ideas 
for  testing.  These  are  playable  scenarios  (2)  that  are  instrumented  with  metrics  of  interest 
(e.g.,  system  selected,  rounds  fired,  speed  attained,  distance  traveled)  specified  by  the 
program  office.  These  scenarios  are  made  available  for  play  via  a  game  server  (3)  that 
allows  distributed  warfighter  players  (4)  to  play,  capturing  diagnostic  information. 
Communities  of  players  are  coordinated  via  conventional  social  media  (5).  Players  can 
modify  scenarios  locally.  ESP  does  not  assume  the  best  scenarios  will  come  from  the 
program  offiee.  More  likely,  the  best  ideas  will  be  modifications  of  those  ideas  made  by 
players.  Game-play  diagnostic  data  (6)  and  modified  scenarios  (7)  are  returned  to  the 
server  and  then  to  the  program  office  (8).  Program  offices  can  also  interact  with  players 
via  the  community  (9).  Science  and  technology  programs  (10)  will  use  ESP  to  test  new 
ideas  at  minimal  expense  before  follow-on  larger  investment.  They  will  also  use  ESP  to 
demonstrate  concepts  to  potential  transition  customers  (11)  to  obtain  buy-in  early,  which 
will  facilitate  an  increase  in  successful  transition  to  programs  of  record. 


Scenarios  fguostions,  models,  measures) 


Transition 


/  -  / 

/ 

‘  Science  &  Technology 


Gamification 


Figure  21 .  ESP  Schematic  (from  Murray  et  ah,  2014) 
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APPENDIX  D.  EARLY  SYNTHETIC  PROTOTYPING  TRIAL  GAME 

SESSION  SCRIPT 


Welcome  Comments:  Thank  you  for  agreeing  to  participate  in  our  study.  Introductions. 

The  purpose  of  the  study  is  to  determine  how  game  environments  might  be  used  to  explore 
novel  design  concepts  early  in  the  acquisition  cycle. 

The  Army  is  calling  this  concept  Early  Synthetic  Prototyping. 

For  our  study,  we  will  be  using  a  robotic  vehicle  called  Wingman.  Wingman  can  be 
equipped  with  a  variety  of  weapons,  armor,  and  engines  that  changes  the  firepower, 
vulnerability,  and  speed  and  agility  of  the  vehicle.  We  will  be  using  some  pre-configured 
versions  in  our  testing  and  they  are  referred  to  as  autonomous  platform  demonstrators 
(APDs). 

We  are  going  to  start  with  a  familiarization  scenario  so  that  everyone  can  become 
comfortable  with  how  the  keyboard  and  mouse  are  used  to  control  your  character  and 
vehicle.  Once  everyone  is  comfortable,  we  will  start  the  three  main  scenarios.  For  each  of 
the  three  scenarios,  there  will  be  a  blue  team.  For  the  second  and  third  scenarios,  there  will 
also  be  a  red  team.  The  blue  team  will  have  a  specific  mission  objective  and  will  be  given 
access  to  one  or  more  Wingman  vehicles  with  which  to  accomplish  that  mission.  The  red 
team’s  objective  is  always  the  same — destroy  the  blue  team. 

Before  each  mission,  we  will  read  you  the  mission  parameters  so  you  know  specifically 
what  you  are  asked  to  do.  We  will  separate  red  from  blue  so  that  each  team  can  privately 
discuss  how  they  intend  to  accomplish  their  goals.  When  both  teams  are  ready,  we  will 
begin  the  scenario.  We  are  recording  the  sessions  so  please  talk  aloud  so  we  can  hear  what 
you  say.  We  can  restart  a  scenario  at  any  time  or  we  can  repeat  a  scenario  if  you  wish  to  try 
it  again.  During  a  scenario,  we  can  answer  any  question  regarding  the  game  interface  but 
we  will  not  answer  questions  or  hint  about  how  to  win  the  scenario. 

We  want  you  to  play  as  much  and  as  often  as  you  would  like.  We  can  extend  today’s 
session  if  time  permits  or  have  you  return  for  another  opportunity  to  play.  Please  be  patient 
as  you  and  your  peers  try  to  learn  the  game;  it  will  be  challenging  if  you  limited  game 
experience.  And,  it  will  be  different  than  other  games  you  have  played  before. 

Before  and  after  each  scenario,  we  will  pause  ask  you  some  questions.  We  request  that  you 
wait  for  instructions  while  in  the  game  to  ensure  we  are  set  to  proceed. 

Remember,  Wingman  is  a  concept.  It  doesn’t  really  exist.  Assume  the  Wingman  Program 
Manager  is  sitting  right  here. 

What  would  you  tell  him  about  the  vehicle  itself  or  how  you  would  use  it? 
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Are  there  missions  we  did  not  explore  here  for  whieh  you  think  it  might  be  an  effeetive 
asset?  This  study  is  about  whether  game  play  can  inform  the  early  acquisition  process. 

Tell  us  what  you  think. 

<break> 

Informed  Consent 


Before  we  begin,  we  need  to  ask  that  you  read  and  sign  this  informed  consent  form  which 
states  that  you  are  voluntarily  participating  and  that  we  can  record  what  you  say  and  do  for 
the  purposes  of  this  study  only. 

DLI:  Army  Behavioral  Health  Clinic  contact  info:  (831)  242-4328 

Questionnaire  [Demographics  at  the  beginningl 

Next,  please  fill  out  this  short  demographic  questionnaire  so  that  we  know  a  little  about 
your  experiences  and  game  playing  history. 


In  VBS2 

Basic  game  navigation: 

The  VBS2  Platform  is  used  to  provide  a  first  person  shooter  gaming  environment 
in  which  to  demonstrate  employment  of  the  various  APDs  in  different  terrain  and  tactical 
employment  scenarios.  The  first  scenario  gives  the  opportunity  for  user  familiarization. 
More  specific  information  on  each  scenario  follows  in  the  subsequent  section. 

VBS2  -  Infantry  Controls  Walk  through  keyboard  map 
Movement  keys 

W:  Forward;  W+Shift:  Run;  W+W:  Short  Sprint 

S:  Backward 


A:  Left 

D: 

Right 

Weapons  Keys 

R:  Reload 

V: 

Optics  view 

I:  Open  gear  menu 

Control  +shft:  on/off  safe 

+/-: 

Zoom  in/out 

VBS2  -  Vehicle  Controls  (See  Reverse 
+/-:  Zoom  in/out 

of  keyboard  map) 

Enter:  3d  Person  View  Tactical  View 
N:  Vision  Mode  (Important  for  visibility) 

L:  Lights  ON/OFF  (Important  for  visibility) 

U:  To  enter  the  APD  view  and  navigate  the  ADP 
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Aim  the  Turret  using  the  mouse;  Fire  the  APD  mounted  weapon  with  the  mouse  (Left 
Cliek) 

Scroll  wheel:  scroll  through  menu  of  available  options  to  interact  with  the  vehicle 
Esc:  to  reach  option  to  abort  scenario 


APD  Options: 

APDs  are  outfitted  with  a  level  of  armor,  a  weapon  or  weapons,  a  set  engine  power,  and  a 
maximum  speed.  These  four  factors  can  be  weighed  to  determine  which  type  of  APD  is 
more  appropriate  for  the  job.  During  today’s  trial  you  will  be  able  to  employ  the  follow 
APDs: 

APD  25mm  /  7.62 
APD  Ml 34 
APD  Speedy 
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Autonomous  Platform  Demonstrators 

(APD) 


APD  25mm  /  7.62mm 
-Armor;  200 

-  Weapors:  M240,  M242  Bushmaaer 
-Engine  Power:  600 

-Max  Speed:  80 

APD  Speedy 
•  Armor.  100 

-  Weapons:  M240 

-  Engine  Power;  2000 
-Max  Speed;  40 

APD  Balanced 

-  Armor;  200 

-  Weapons;  M240 
-Engine  Power;  600 
-Max  Speed  SO 

APD  Heavy 
-Armor;  300 

-  Weapons;  M240 
-EnginePovrer;400 

-  Max  Speed;  60 

APD  .SOCal 
-Armor  200 

-  Weapons;  M2  Browning 
-EnginePower;600 
-Max  Speed;  80 

APD  Mkl9 
-Armor;  200 

-  Weaports:  MK19 
-Engine  Power:  600 
-Max  Speed:  80 


APD  M134 

-  Armor  200 

•  Weapons;  M134 
-Engine  Power;  600 
•Max  Speed  80 

APD  ARAS  50 

•  Armor;  600 

-  Weapons:  Lethai/Nonlethal  50  bal 
-Engine  Power;  800 

•Max  Speed:  70 

APD  SSADT 

-  Armor:  600 

-  Weapons:  Aaive  Denei  System 
-Engine  Power;  800 

-Max  Speed  70 

APD  ARAS  7.62 

-  Armor;  600 

-  Weapons;  Lechal/Nonleihal  7.62  bal 
•Engine  Power;  800 

•Max  Speed;  70 

APD  TOW 

-  Armor:  200 

-  Weapons;  TOW  Launcher,  Heilfire Launcher 

-  Engine  Power;  600 
•Max  Speed;  80 

APD  •  Boomerang 
-Armor;  200 

-  Weapons;  TOW  Laurtcher,  Heilfire  Launcher 
-Engine  Power;  600 

•Max  Speed;  80 
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Training  scenarios  -  training  navigation 
Click  on  APD  Navigation 

This  can  be  run  multiple  times  if  needed  for  all  players  to  beeome  eomfortable. 

Cliek  Start 

Basie  weapons  training  requires  you  to: 

Gain  eontrol  of  an  APD  remotely. 

Navigate  through  an  obstaele  eourse  and  eliminate  any  targets  eneountered. 

Double  eliek  on  the  notepad  at  the  top  like  the  spirals  to  reduee  its  size. 

Seroll  using  the  mouse  wheel  to  zoom  in  or  out  of  porto. 

Beeause  this  is  training,  the  map  does  not  show  your  objeetives. 

This  training  seenario  is  not  networked  so  others  won’t  be  visible. 

Cliek  eontinue 

Cliek  M  to  reaeh  the  map  to  get  oriented  to  where  you  are  in  Porto. 

M  again  will  return  you  to  the  seenario 

Look  for  the  yellow  eue  on  the  sereen  to  take  you  to  your  first  objeetive. 

Piek  up  the  AT-4:  engage  the  HMMWV  with  the  AT-4  or  the  M-4 
Use  V  to  site  in  and  Left  eliek  to  fire 

Seroll  wheel  get  APD  eontroller 

Seleet  an  APD;  Choose  APD  134  or  APD25mm/7.62 

Hit  U  to  get  the  APD  view. 

Seroll  to  sereen 

For  ease  of  navigation  we  reeommend  keeping  the  weapon  barrel  between  the  traek  to  ensure  your 
visibility  matehes  your  movement;  Your  view  will  always  be  where  you  weapon  is  direeted. 

Hit  the  +/-  keys  to  zoom  in  or  out. 

In  the  training  navigation,  follow  the  yellow  arrow  eues  to  meet  your  training  objeetives. 

If  you  “dismount  your  APD”  and  return  to  your  eharaeter  you  will  start  over. 

Questions: 

Do  you  want  to  play  again? 

How  do  you  think  you  did? 

What  do  you  think  you  eould  do  to  improve  your  performanee? 

We  ean  eontinue  playing  or  set  up  another  session. 
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Scenario  1  -  APD  Chase  (Ensure  server  is  set  to  Takistan  and  Chase) 


Go  to  networking;  There  is  one  network  game  session  available. 

Seleet  the  player  that  eorresponds  to  your  station  number. 

Take  note  of  the  verbal  orientation  provided  next  to  the  role. 

Ex.  1-1-A-!  POS  Hillside,  south  of  main  road 
Bluel  select  player  BLU  1,  etc. 

Now  that  everyone  has  the  APD  Chase  scenario  up  on  their  screens,  let’s  take  a  moment  to  get 
oriented.  The  scenario  and  objective  is  (Read  objective  for  APD  Chase): 

Scenario:  Recent  Intel  suggests  A  HIGH-RANKING  OPFOR  OFFICER  is  currently  in  the  area  of 
Takistan,  although  his  exact  whereabouts  is  unknown.  A  convoy  in  the  small  village  of  Hazar  Bagh  is 
set  to  meet  him.  Using  your  APDs,  follow  the  convoy  without  being  seen.  When  contact  with  the 
target  is  made,  eliminate  him,  all  units  supporting  him,  and  the  convoy. 

Click  the  arrow  on  the  notepad  to  see  objectives  to  meet  for  game  success: 

a.  Remain  under  cover  and  avoid  detection. 

b.  Eliminate  OPRFOR  once  contact  with  the  target  is  made 

c.  No  APDs  can  be  destroyed. 

Find  your  APD  on  the  screen:  It  should  correspond  to  your  station  number. 

a.  Click  on  the  hyper-linked  name  on  the  notepad 

b.  Zoom  in  and  out  by  scrolling  the  mouse  wheel 

c.  For  a  larger  view,  minimize  the  notepad  by  double-clicking  near  the  top  spiral. 

d.  You  can  manipulate  the  map  view  to  get  a  better  orientation  by  sliding  the  mouse. 

Feel  free  to  discuss  the  scenario  and  work  a  plan  among  your  team  (3-5  min). 

We  are  interest  in  your  estimate  of  the  situation  and  a  brief  outline  of  your  plan  to  meet  the  objective. 

-  With  the  limited  information  available  to  you,  what  is  your  estimate  of  the  situation? 

-  What  is  your  plan: 

How  do  you  think  you  ’ll  employ  your  APDs? 

How  will  you  employ  your  characters? 

All  players  hit  continue;  Then,  server  hits  continue 

Scroll:  select  UGV  Controller  -  This  is  your  person  view 
U:  To  enter  the  perspective  of  your  APD 

Scroll:  Select  Screen — scroll  again  to  change  optics 
N:  Optics  view;  F:  Fights 

Click  continue.  (Click  continue  on  the  server  to  start  the  scenario) 

[Server:  exit  out  and  start  a  new  host  session] 
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Scenario  2  -  APD  Attack  (Ensure  server  is  set  to  Geotypical  Eastern  Europe  and  Attack) 


Two  Red  Players  will  move  to  LabA. 

Go  to  networking;  There  is  one  network  game  session  available  for  players. 

Seleet  the  player  that  eorresponds  to  your  station  number. 

Take  note  of  the  verbal  orientation  provided  next  to  the  role. 

Ex.  1-1-A-!  POS  Hillside,  south  of  main  road 
Bluel  seleet  player  BLU  1,  ete. 

BLUFOR  Objeetive  You  must  rescue  the  prisoners  from  the  OPFORs,  but  the  camp  is  heavily 
guarded.  You  each  have  an  APD,  but  most  of  the  EN  are  armed  with  rocket  launchers,  so  a  direct 
assault  will  not  work.  The  prisoners  will  start  running  to  freedom  once  the  shooting  begins,  but  the 
OPFOR  will  shoot  at  them  as  they  run  away. 

Rescue  as  many  prisoners  as  possible.  Eliminate  all  of  the  OPFORs. 

OPFOR  Objective  Two  BFFIFOR  operatives  with  APDs  are  attacking.  You  must  stop  them  from 
freeing  the  prisoners,  but  do  not  shoot  the  prisoners,  even  if  they  have  escaped. 

Eliminate  both  of  the  BFFlFORs 

Feel  free  to  discuss  the  scenario  and  work  a  plan  among  your  team  (3-5min). 

We  are  interest  in  your  estimate  of  the  situation  and  a  brief  outline  of  your  plan  to  meet  the  objective. 

-  With  the  limited  information  available  to  you,  what  is  your  estimate  of  the  situation? 

-  What  is  your  plan: 

How  do  you  think  you  ’ll  employ  your  APDs? 

How  will  you  employ  your  characters? 

Click  continue. 

(Click  continue  on  the  server  to  start  the  scenario) 

Ml  34  does  not  have  a  long  barrel;  recommend  zooming  out  to  see  perspective  of  APD. 
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Scenario  3  -  APD  Defend  (Ensure  server  is  set  to  Porto  and  Defense) 


Two  Red  Players  will  move  to  LabA. 

Go  to  networking;  There  is  one  network  game  session  available  for  players. 

Seleet  the  player  that  eorresponds  to  your  station  number. 

Take  note  of  the  verbal  orientation  provided  next  to  the  role. 

Ex.  1-1-A-!  POS  Hillside,  south  of  main  road 
Bluel  seleet  player  BLU  1,  ete. 

BLUFOR  Objeetive  Defend  your  position  from  OPFOR  units  for  5  minutes.  Each  of  you  have  an  APD  under 
your  control.  If  the  APD  is  destroyed,  you  will  lose  5  points  and  the  APD  will  automatically  respawn  in  its 
original  positions.  If  either  player  dies,  you  will  lose  10  points  and  the  enemies  will  all  be  reset  to  their  original 
positions.  Rescue  as  many  prisoners  as  possible. 

OPFOR  Objective  Seek  and  destroy  the  BLUFOR  units  and  vehicles  for  5  minutes.  Every  time  a 
BLUFOR  player  gets  killed,  you  will  be  respawned  in  one  of  your  three  start  positions.  If  a  BLUFOR 
APD  gets  destroyed,  it  will  respawn  at  its  start  position.  If  you  die,  you  will  automatically  get 
respawned  on  one  of  your  starting  positions. 


Feel  free  to  discuss  the  scenario  and  work  a  plan  among  your  team  (3-5min). 

We  are  interest  in  your  estimate  of  the  situation  and  a  brief  outline  of  your  plan  to  meet  the  objective. 

-  With  the  limited  information  available  to  you,  what  is  your  estimate  of  the  situation? 

-  What  is  your  plan: 

How  do  you  think  you  ’ll  employ  your  APDs? 

How  will  you  employ  your  characters? 


Click  continue. 

(Click  continue  on  the  server  to  start  the  scenario) 
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Post  -  Run  verbal  group  interview 


Is  anyone  feeling  any  adverse  impacts  from  playing  the  game? 

If  yes,  please  feel  free  to  remain  behind  and/or  contact  the  DLI  Behavioral  Health  Clinic  at 

(831)  242-4328. 

ESP  Post-Run  Interview  Outline 
WINGMAN  SPECIFIC 

1 .  What  general  observations  and  recommendations  do  you  have  about  the  Wingman 
concept? 

a.  Is  it  useful?  If  so,  for  which  scenarios?  Why? 

b.  Should  it  be  configured  differently?  Scenario  dependent? 

2.  How  would  you  recommend  deploying  Wingman? 

a.  For  which  missions? 

b.  In  which  configuration? 

i.  Firepower  /  Armor 

ii.  Agility  /  Range 

c.  How  many? 

d.  Who  operates  them? 

3.  What  can  or  would  you  do  differently  with  Wingman  that  you  might  not  do  with  a 
person? 

4.  What  concerns  you  most  about  the  Wingman  concept?  (What  are  the  greatest 
barriers  to  successful  development  and  deployment?) 

5.  What  are  the  greatest  opportunities  that  Wingman  might  offer? 

a.  New  missions? 

b.  Radically  different  doctrine?  (e.g.  all  robots  and  no  people) 

ESP  SPECIFIC 

1 .  Is  a  game  environment  a  suitable  place  to  experiment  with  early  ideas  like 
Wingman? 

2.  Was  the  game  environment  “real”  enough  that  you  could  explore  new  ideas? 

3.  If  we  provided  an  editor  that  allowed  you  to  reconfigure  the  vehicle  or  the 
scenario,  would  that  add  to  your  interest? 

4.  What  conclusions  about  Wingman  do  you  think  you  can  draw  from  playing  a 
game? 

5.  What  conclusions  about  Wingman  do  you  NOT  think  you  can  draw  from  playing 
a  game? 

6.  Do  you  think  you  can  have  an  impact  on  how  the  military  tactically  employs  its 
personnel  or  sets  its  equipment  specifications? 
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Red  1  Red  2 


APPENDIX  E.  EARLY  SYNTHETIC  PROTOTYPING  TRIAL  GAME 

SESSION  LAB  LAYOUT 


ESP  Game  Session  Layout 


LabB 


1  MOVES  Lab  7 

MOVES  Lab  8  | 

Bluel 

Blue2 

MOVES  LablO 

MOVES  Labll 

MOVES  Lab  4  | 

i 

Server 

1 

Blue3 

Blue4 

Lab  A 


=  audio  recording  device 
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APPENDIX  F.  EARLY  SYNTHETIC  PROTOTYPING  GAME  TRIAL 

PRE-DEMOGRAPHIC  SURVEY 

EARLY  SYNTHETIC  PROTOTYPING  PARTICIPANT  QUESTIONNAIRE 

1 .  What  branch  of  service  are  you  affiliated  with? _ 

2.  What  is  your  status  (cirele  one)?  Civilian  (skip  to  c)  /  Active  Duty  /  Reservist 

a.  What  is  your  rank?  How  many  years  of  service  do  you  have? 

_ / _ 

b.  What  is  your  MOS  /  Branch  /  Rate  /  Community  designator? 

3.  Do  you  have  prior  video  gaming  experience  within  the  past  two  years? 

Yes  /  No 

a.  How  mueh  (cirele  one)? 

<  1  hour  per  week 

1- 2  hours  per  week 

2- 5  hours  per  week 
>  5  hours  per  week 

b.  Is  your  gaming  experience  with  first  person  shooter  games?  Yes  /  No 

1 .  List  a  few  of  your  favorite  games  (of  any  type) 

a.  _ 

b.  _ 

c.  _ 

d. 
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APPENDIX  G.  EARLY  SYNTHETIC  PROTOTYPING  GAME  TRIAL 

POST-PLAY  SURVEY 


Participant  Survey 

Knowing  that  your  participation  contributes  to  the  Early  Synthetic  Prototyping  concept 
and  the  Wingman  project,  do  you  feel  that  your  time  was  spent  effectively  today? 
Effective  semi-effective  Neutral  semi-ineffective  Ineffective 

1  2  3  4  5 

If  yes,  how?  If  no,  why  not? 


What  kinds  of  ideas  do  you  have  for  future  concepts  or  equipment? 


future  tactics? 


future  doctrine? 


Other  ideas? 


Would  you  contribute  to  future  efforts  to  develop  or  test  ideas  in  a  game  environment? 

Likely  Somewhat  Likely  Neutral  Somewhat  Unlikely  Unlikely 

1  2  3  4  5 

If  no,  why  not? 


Would  you  encourage  others  to  participate  in  future  efforts  to  develop  or  test  ideas  in  a 
game  environment? 

Likely  Somewhat  Likely  Neutral  Somewhat  Unlikely  Unlikely 

1  2  3  4  5 


Do  you  have  any  ideas,  recommendations,  or  suggestions  for  Early  Synthetic 
Prototyping? 


Do  you  have  any  ideas,  recommendations,  or  suggestions  for  Wingman? 


Would  you  play  this  game,  or  one  very  similar  to  it  again? 
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YorN 


How  often  would  you  play  this  game  if  you  had  access? 

Access  via  download  to  a  personal  device:  # _ times  per  week/month/yr 

Access  via  the  internet:  # _ times  per  week/month/yr 

Access  and  time  at  work:  # _ times  per  week/month/yr 

Wingman  Autonomous  Platform  Demonstrators  (APD) 

Mobility: 

What  is  your  opinion  on  the  mobility  of  the  APD? 

Effective  semi-effective  Neutral  semi-ineffective  Ineffective 

1  2  3  4  5 


How  would  you  modify  the  APD’s  mobility? 


Armament: 

What  is  your  opinion  of  the  APD’s  armament? 

Effective  semi-effective  Neutral  semi-ineffective  Ineffective 

1  2  3  4  5 


How  would  you  modify  the  APD’s  armament? 


Optics: 

What  is  your  opinion  of  the  APD’s  optics? 

Effective  semi-effective  Neutral  semi-ineffective  Ineffectiv^e 

1  2  3  4  5 


How  would  you  modify  the  APD’s  optics? 


VBS2  employment? 

Were  you  able  to  navigate  the  APD  within  the  scenario? 


What  challenges  or  positive  attributes  do  you  have  regarding  the  movement  and  views 
within  the  game  environment? 


Do  you  think  the  APD  is  a  realistic  asset  for  employment  by  the  military?  Yes  or  No? 
Why? 
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Scenario  Feedback  -  NAVIGATION 


yes  most  some  few  none 

Did  you  understand  the  scenario  objectives?  1  2  3  4  5 

Comments: 


yes  most  some  few  none 

Did  you  meet  the  scenario  objectives?  1  2  3  4  5 

Comments: 


Would  you  play  this  scenario  again  if  given  the  opportunity? 
yes  most  some  few  none 

1^  2  3  4  5 

Comments: 


Did  the  game  environment  (VBS2)  make  it  easy  or  difficult  to  reach  the  objectives? 

Easy  Somewhat  Easy  Neither  Somewhat  difficult 

Difficult 

1  2  3  4  5 

Comments: 


How  did  you  communicate  with  your  teammate  in  this  scenario,  was  it  effective?  Would 
you  change  your  communication  strategy? 


Did  this  scenario  and  the  game  environment  give  you  the  opportunity  to  produce  a 

creative  solution  to  meet  your  objective? 

yes  most  some  few  none 

1  2  3  4  5 

Comments: 
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Scenario  Feedback  -  APD  CHASE 


yes  most  some  few  none 

Did  you  understand  the  scenario  objectives?  1  2  3  4  5 

Comments: 


yes  most  some  few  none 

Did  you  meet  the  scenario  objectives?  1  2  3  4  5 

Comments: 


Would  you  play  this  scenario  again  if  given  the  opportunity? 
yes  most  some  few  none 

1^  2  3  4  5 

Comments: 


Did  the  game  environment  (VBS2)  make  it  easy  or  difficult  to  reach  the  objectives? 

Easy  Somewhat  Easy  Neither  Somewhat  difficult  Difficult 

1  2  3  4  5 

Comments: 


How  did  you  communicate  with  your  teammate  in  this  scenario,  was  it  effective?  Would 
you  change  your  communication  strategy? 


Did  this  scenario  and  the  game  environment  give  you  the  opportunity  to  produce  a 

creative  solution  to  meet  your  objective? 

yes  most  some  few  none 

1  2  3  4  5 

Comments: 
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Scenario  Feedback  —  APD  Attack 


yes  most  some  few  none 

Did  you  understand  the  scenario  objectives?  1  2  3  4  5 

Comments: 


yes  most  some  few  none 

Did  you  meet  the  scenario  objectives?  1  2  3  4  5 

Comments: 


Would  you  play  this  scenario  again  if  given  the  opportunity? 
yes  most  some  few  none 

1^  2  3  4  5 

Comments: 


Did  the  game  environment  (VBS2)  make  it  easy  or  difficult  to  reach  the  objectives? 

Easy  Somewhat  Easy  Neither  Somewhat  difficult  Difficult 

1  2  3  4  5 

Comments: 


How  did  you  communicate  with  your  teammate  in  this  scenario,  was  it  effective?  Would 
you  change  your  communication  strategy? 


Did  this  scenario  and  the  game  environment  give  you  the  opportunity  to  produce  a 

creative  solution  to  meet  your  objective? 

yes  most  some  few  none 

1  2  3  4  5 

Comments: 
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Scenario  Feedback  —  APD  Defend 


yes  most  some  few  none 

Did  you  understand  the  scenario  objectives?  1  2  3  4  5 

Comments: 


yes  most  some  few  none 

Did  you  meet  the  scenario  objectives?  1  2  3  4  5 

Comments: 


Would  you  play  this  scenario  again  if  given  the  opportunity? 
yes  most  some  few  none 

1^  2  3  4  5 

Comments: 


Did  the  game  environment  (VBS2)  make  it  easy  or  difficult  to  reach  the  objectives? 

Easy  supported  effort  somewhat  Nei&er  supported  or  prevented  somewhat  Etifficult  prevented  effort 

1  '  2  3  4  5 

Comments: 


How  did  you  communicate  with  your  teammate  in  this  scenario,  was  it  effective?  Would 
you  change  your  communication  strategy? 


Did  this  scenario  and  the  game  environment  give  you  the  opportunity  to  produce  a 

creative  solution  to  meet  your  objective? 

yes  most  some  few  none 

1  2  3  4  5 

Comments: 
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APPENDIX  H.  SUMMARY  TRANSCRIPTION  OF  POST-GAME 

INTERVIEW  (PILOT) 


Pilot: 

Bounding  overwatch  for  the  chase  scenario;  lights  were  problematic  for  networked 
scenario. 

Concern  expressed  over  whether  a  tracked  vehicle  was  appropriate  for  this  mission. 
Attack: 

Blue  Plan/execution — anticipated  a  wall  around  the  prison.  Intended  to  rendezvous  on  the 
north  side.  AI  took  over  the  APD  and  conducted  an  unintended  frontal  attack. 

Red  Plan/execution — Intended  to  take  out  the  operators  rather  than  the  APDs.  Player  was 
taken  out  because  he  was  unable  to  move  quickly  enough  in  game  environment  to  aim, 
shoot,  and  move  to  safety.  Other  red  player  noticed  the  smoking  APD  and  sought  the 
operator  but  was  noticed  by  the  BLUFOR  first.  Wounded,  he  wasn’t  able  to  move. 
Noticed  the  APD  moving  around  the  perimeter  and  “hid”  by  a  tree.  Shot  RPG  and  AK 
rounds.  Saw  the  possibility  to  employ  APD  as  a  vehicle  home  improvised  explosive 
device  (VBIED)  or  mobile  mine.  “Dead”  comrade  was  able  to  observe  the  movement  of 
the  BLUFOR  so  he  knew  approximate  location.  Red  player  was  successful  with  a  long 
distance  shot.  Small  arms  were  the  deciding  factor 
Defend: 

Red  plan — ^whoever  was  spawned  initially  in  town  would  support  by  fire  long  range. 
They  didn’t  realize  they  would  have  AI  players.  Re-spawn  was  problematic  because  in 
the  middle  of  aiming,  the  player  would  re-spawn  to  origin.  The  situation  dissolved  to 
chaos  because  they  weren’t  able  to  maintain  aim  and  make  effective  use  of  their  shots. 
Blue  plan — eyes  in  multiple  directions  to  observe  full  fields  of  view.  Players  were  hiding 
in  grass  behind  the  compound  in  an  attempt  to  maintain  safety. 

Which  scenarios  did  they  like? 

Chase.  Compared  to  an  unmanned  aerial  vehicle  (UAV)  recon  and  give  Predator  permission  to 
shoot.  The  Red  team  also  liked  the  Attack  scenario  because  they  were  successful.  Blue  like  APD 
Chase.  Both  teams  did  not  like  the  defend  scenario  due  to  the  constant  re-spawn  and 
disorientation. 

Vehicle? 

Wheeled  vehicle  appeared  like  a  normal  tactical  vehicle.  Situational  awareness  of  whether  lights 
were  on  or  off.  The  menu  interfered  with  the  observation  of  the  screen  command. 

Recommend  simple  radio  buttons  present  along  the  border  of  the  screen  to  indicate  on/off  and  be 
able  to  manipulate  the  state  as  well.  The  scroll  wheel  was  a  challenging  way  to  find  out  the  “state 
of  the  system.” 

Training  didn’t  give  all  the  screen  options.  (This  was  later  corrected  by  providing  additional 
direction  during  the  familiarization  scenario.) 

Recommend  drop  waypoints  to  allow  vehicle  to  auto-drive  to  a  particular  location  and  then  the 
operator  could  gain  situational  awareness  by  observing  through  the  turret  rather  than  have  to 
navigate  and  observe  simultaneously.  Having  a  decoupled  turret  view  is  spatially  disorienting. 
Drive  mode — lock  the  gun  position  to  the  front.  Big  screen  becomes  drive  screen  and  the  small 
screen  becomes  the  turret  screen. 
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Two  operators  recommended:  one  operator  for  the  robot,  one  soldier/Marine.  Comparison  made  to 
Halo. 

Treat  this  like  a  UAV — doesn’t  need  an  operator  adjacent  to  it  as  long  as  we  are  capable  of 
communicating  with  it.  There  could  be  two  operators  with  one  person  navigating  and  moving  and 
the  other  able  to  target  and  make  use  of  observations.  Waypoints  would  be  useful.  Trees  and 
obstacles  presented  challenges  to  movement  and  observation.  Player  questions  robustness  of 
vehicle  considering  environmental  challenges. 

Toggle  between  views — drive  view  or  turret  view  based  on  the  task  to  be  accomplished.  Navigate, 
drive,  aim,  shoot  is  challenging  for  a  single  person. 

Players  would  prefer  a  game  pad  rather  than  keyboard. 

Personalized  auto-settings  like  “pre-mission,”  “tactical-night-mode,”  “tactical-day-mode,”  “non- 
tactical-night,”  “non-tactical-day,”  etc.  to  avoid  the  situational  disorientation  involved  in  coming 
into  the  scenario. 

Example  of  turret  indicator  feature  for  tanks  in  relation  to  the  hull.  (This  is  a  feature  available  on 
different  APD  versions.) 

Operator  is  not  just  going  to  fall-in  on  a  piece  of  equipment.  The  operator  would  be  far  more 
familiar  with  the  system’s  capabilities  and  able  to  tweak  the  settings  prior  to  embarking  on  the 
mission. 

“Tactical-day”  and  “tactical-nighf’  settings  so  the  players  could  customize  from  there. 

Zoom  and  perspective — recommended  farthest  out  zoom. 

What  kinds  of  tactical  tasks  could  you  accomplish  with  wingman? 

How  much  do  you  trust  it?  Performance.  If  there  is  another  controller  in  the  vicinity  that  the 
human  can  interact  with  the  trust  level  would  be  higher  than  an  operator  out  of  sight  with  whom 
you  could  lose  communication  or  a  fully  automated  system.  Fear  of  hacking. 

Armored  platform  with  gun:  support  by  fire,  reconnaissance.  Not  concerned  about  it  being  killed. 
Not  as  concerned  about  destruction  or  capture. 

Perception  that  capability  is  in  line  with  what  a  human  can  currently  provide.  An  operator  could 
enhance  the  capability  but  skeptical  of  an  autonomous  vehicle. 

Comparison  to  UAVs — trust  from  UAVs  grew  in  Iraq  significantly.  Compared  to  Ravens  and 
Shadow — initially  perception  was  that  the  UAVs  only  saw  a  small  picture.  Current  perspective  is 
more  trusted  based  on  operator  competency.  Zooming  and  movement  depends  on  the  operator 
skill  level.  Having  it  operated  remotely  wouldn’t  be  too  much  of  an  issue  but  there  would  be 
challenges  with  plans,  contingencies,  and  communications.  Preference  for  the  operator  to  be  closer 
to  the  supported  unit/soldiers. 

Would  a  game  session  inform  ESP? 

Recommend  incorporating  the  more  specific  user  base. 

Challenges  existing  doctrine  by  incorporating  non-standard  user  base  to  evaluate  and  try  variant 
tactical  employment. 

May  be  a  tendency  for  experienced  users  to  employ  the  asset  in  accordance  with  their  experiences. 
Recommend  large  scale  study  to  achieve  greater  consensus  than  small  isolated  sessions.  Increase 
observations  longitudinally  to  capture  proficiency  gains  with  the  gaming  environment.  This  would 
reduce  the  division  between  naive  and  more  proficient  gamers  and  bring  ideas  from  naive  and 
experienced  tacticians  into  the  same  environment. 

This  is  what  it  should  do — what  else  can  it  do? 

Clear  questions — what  specific  features  are  we  looking  to  evaluate.  What  was  the  interface  like, 
how  could  it  be  improved. 
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VBS2  is  available.  Networked  opportunity  may  present  challenges  to  collecting  data.  However, 
the  ability  to  reach  more  players  and  attempt  to  achieve  consensus. 

Tactics?  Use  this  environment  to  develop  new  tacties? 

Re-spawn  scenario  should  keep  people  “dead”  so  they  know  the  repercussions  of  their  actions. 

Play  ten  rounds:  Play  until  players  are  dead,  take  a  break  after  each  round  to  rehash,  assess  the 
plan,  play  again,  repeat.  Recommend  overhead  view  for  the  mini-AAR. 

Opportunity  to  select  terrain  and  modify  vehicles?  Would  that  grab  players’  attention  and 
incite  participation? 

Ability  to  modify  allows  players  to  partake  but  then  players  might  figure  out  ways  to  “game”  the 
game  and  become  indestructible? 

What’s  the  aim — entertainment  or  practical  application?  Need  to  balance. 

Conversation  on  incorporation  of  real  world  locations — geotypical  (not  geo-specific)  is 
mostly  accurate  in  VBS2. 
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APPENDIX  I.  SUMMARY  TRANSCRIPTION  OF  POST-GAME 

INTERVIEW  (SESSION  1) 


Group  1 

-Enjoyed  the  “hunting”  aspect  of  the  attack  scenario.  There  was  time  available  in  the 
scenario  to  get  oriented  and  make  his  way  around.  The  defend  scenario  was  too  frantic 
Liked  APD  Attack — prison.  Didn’t  like  APD  defend  due  to  getting  disoriented 
frequently. 

Red  force  attempted  to  go  around  the  backside  but  they  kept  getting  respawned  (we  think 
this  was  also  due  to  the  team  trying  to  go  outside  of  the  “bubble”  of  the  game).  Saw  it  as 
a  penalty  when  they  took  out  the  enemy. 

APD  Attack — prison.  Liked  the  terrain  and  being  able  to  take  various  positions  and  be 
able  to  hide. 

Recommendations  for  wingman? 

Insurgents  didn’t  have  technology. 

Seems  like  players  are  less  connected  so  the  players  were  more  willing  to  put  themselves  out 
there.  In  the  APD  chase  they  kept  getting  caught  because  they  weren’t  cautious  and  kept  getting 
too  close.  In  APD  attacked,  tried  to  get  close  to  get  a  good  look  but  the  APD  was  quickly  taken 
out. 

Easy  target  but  easier  to  make  bold  moves. 

One  player  had  major  frustration  with  navigation  and  controls.  The  terrain  presented  a  huge 
problem. 

Can’t  turn  your  head  in  the  truck. 

You  can  turn  the  turret  so  you  could  drive  and  look  outside  of  your  field  of  view. 

Which  scenarios  do  you  think  the  robots  would  be  most  useful? 

Defense.  Can  be  employed  like  a  patrol  to  expand  your  perimeter. 

You  don’t  really  care  that  you’re  putting  it  out  there. 

You  take  fire  but  then  you  get  information  back  on  your  adversary’s  location. 

Vehicle  modifications? 

Scenarios  didn’t  test  the  vehicles  to  the  extent  that  they  are  capable  of 
Recommend  a  group  of  5:  3  infantrymen  and  2  vehicle  controllers. 

Lose  big  picture  awareness  with  one  person. 

Alternative  would  be  1  person  with  a  vehicle  and  1  person  without  within  existing  concept. 
Recommended  mode  of  employment  5 — running  2  robots  at  a  time, 
o  Conventional  employment  recommended: 

■  Fuel,  Cl  IX,  ...  a  small  team  would  not  be  able  to  take  advantage  of  an  APD. 

■  Company  sized  element  with  enough  logistical  back  support. 

■  How  would  you  package  this  for  a  rapid  deployment  force? 

■  Logistical  pain  that  a  foot  soldier  doesn’t  have  to  worry  about. 

■  More  power  in  terms  of  supporting  an  APD. 

■  NO  rockets — loading  would  be  a  challenge  and  armament  would  be  susceptible 
to  blowing  up. 

■  Recommend  armament  along  the  same  lines  as  the  unit  it’s  supporting. 
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■  Mkl9  might  be  challenging  to  employ  ...  or,  put  it  in  defilade  somewhere  to  use 
as  a  fire  support  weapons  without  endangering  your  own  men. 

■  MK134  mini  gun  is  small  but  can  send  a  lot  of  rounds  . . .  but  if  the  gun  breaks 
down,  it  becomes  a  sensor. 

Concerns  about  employment? 

In  defense,  it’s  a  big  target.  Concerned  about  destruction  of  technology  or  not  putting  any  sensitive 
equipment/information  on  the  robot. 

Maybe  a  mechanized  unit  that  could  move  it  and  fix  it. 

Concerns  over  hacking  into  its  network  connection. 

Robot  is  not  a  person.  This  brings  functionality  but  is  this  the  right  way  to  employ  our  assets. 
Multiple  players  were  hesitant  to  allow  for  an  autonomous  vehicle. 

Talked  about  2  people,  each  controlling  an  APD  rather  than  2  people  controlling  one  APD. 

ESP?  Game  environment  for  gathering  ideas? 

Practical  place  to  compile  ideas 
Adapt  scenarios  based  on  weaponry. 

Was  game  environment  real  enough  to  explore  ideas? 

Limited  to  certain  types  of  terrain.  Example,  Afghanistan  is  different  than  the  terrain  in  the 
scenarios. 


Editor? 

Customization —  “hands  down”  because  you  can  create  your  strategy  based  on  the  scenario  with 
the  capabilities. 

o  Example:  Suicide  robot,  or  diversionary  element  well  armored  so  its  survivable  and  can 
take  lots  of  enemy  fire. 

Conclusions  for  wingman  different  from  just  talking? 

Playing  gave  players  opportunity  to  see  and  view  what  it  can  do  and  how  to  make  it  work. 
Cheaper  in  the  virtual  environment. 

Challenges:  simulation  of  dust,  parts,  getting  stuck. 

What  happens  when  the  optics  fail  or  a  battery  fails  or  there  is  a  leak  or  a  track  is  thrown? 
Assumptions  of  simulation. 

Good  to  test  a  possible  theory  but  would  need  real  life  to  verify. 
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APPENDIX  J.  SUMMARY  TRANSCRIPTION  OF  POST-GAME 

INTERVIEW  (SESSION  2) 


Group  2 

Wingman  recommendations? 

Trouble  following  re-spawn  as  an  insurgent. 

Which  scenario  did  you  like? 

Prison — ^both  blue  and  red. 

Convoy — good  but  need  to  know  enemy’s  protection  range.  What  kind  of  optics  they  have. 
Otherwise,  the  situation  kind  of  leads  you  to  follow  them  down  the  road. 

Defaults  to  “undetectable”  situation  with  light  off 

What  other  settings  would  be  helpful? 

Being  able  to  reference  location  and  orientation  without  having  to  jump  to  the  map  and  taking  you 
out  of  the  scenario.  More  like  a  map — top  down  view  in  the  comer  of  your  view  screen. 

Followed  compass  directions  but  had  to  verify  where  we  were  on  the  map. 

Looking  for  a  map  overview  on  the  screen. 

Prison  (Attack)  was  mostly  a  first  person  shooter  scenario.  The  APD  didn’t  really  matter.  Once  the 
APDs  were  shot  by  the  rocket  launchers,  the  players  were  able  to  use  the  trucks. 

Red  was  able  to  observe  the  APDs  easily  over  the  horizon 

Going  back  to  the  Chase  scenario,  it  doesn’t  seem  likely  that  two  people  with  two  APDs  would  be 
able  to  get  into  a  town  full  of  bad  guys  undetected;  they  seem  pretty  loud. 

Were  the  robots  useful  in  any  of  the  scenarios? 

Clumsy  to  use  in  defense.  Needed  to  figure  out  how  to  move  around,  which  way  tracks  and  turret 
were  pointed. 

There  is  an  advantage  to  having  a  25mm  gun  at  your  disposal  but  being  the  guy  in  the  fight  and 
trying  to  control  the  robot  is  mentally  too  much  to  handle. 

For  what  kind  of  missions  would  the  robot  be  useful? 

Red  perspective — didn’t  know  where  shots  were  coming  from.  But  the  Blue  team  reported  that 
they  were  using  small  arms  to  provide  the  harassing  fires. 

Can  you  see  a  scenario  where  you  would  use  wingman? 

Defensive  position. 

Blocking,  outer  perimeter  asset. 

To  hold  ground  without  maneuvering. 

APDs  to  mn  patrols  around  a  perimeter  might  be  useful.  The  controller  is  not  “in  the  fight”  but  is 
able  to  control  the  APD.  Controller  not  exposed.  From  personal  experience,  used  a  UAV  to  mn  a 
patrol  around  the  perimeter. 

Do  you  see  someone  controlling  more  than  one  APD? 

On  the  contrary,  I  think  you  need  more  than  one  person  to  control  each  APD.  With  live 
ammunition,  I  think  you  need  one  driver  and  one  shooter  minimum.  Someone  needs  complete 
situational  awareness.  See  the  risk  as  significant  for  live  ammunition  and  controller  getting 
overwhelmed. 
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Armament?  Agility? 

Once  players  were  able  to  control  it,  it  was  fine.  It  moved  fast  enough. 

If  they  took  one  RPG  round  the  APD  was  done.  Compared  to  an  Abrams  that  can  take  several 
rocket  rounds. 

If  20  year  old  armament  technology  can  withstand  a  rocket  launcher,  the  APD  needs  to  be  able  to 
sustain  the  highest  standard  currently  available. 

-Players  asked:  What  would  it  take  to  roll  the  APD  over?  Not  sure  if  controller  would  be  able  to 
see  what  was  in  his  way  and  what  might  cause  the  APD  to  roll  over.  Incline  may  be  disorienting, 
the  user  might  not  be  able  to  see  the  incline  through  the  screen  view. 

Concerns  over  wingman  eoneept? 

Lack  of  realism.  Develop  indifference  to  rounds  coming  at  you. 

Player  responds  differently.  Player  would  get  down  if  incoming  rounds. 

Players  are  more  bold  and  reckless  because  they  know  it’s  not  real. 

Removes  some  natural  inhibitions  in  the  game  environment. 

Red  plan  was  to  draw  fires  because  he  knew  he  would  get  re-spawned. 

Opportunities  for  wingman? 

Employ  it  like  a  UAV  was  mentioned  previously. 

VSO  positions — village  stability  operations.  Supports  a  small  team  of  guys  with  flexibility.  They 
could  do  route  patrols.  Could  be  emplaced  outside  of  perimeter.  If  you  have  the  means  to  recover 
it  or  protect  it,  you  could  employ  it  on  a  piece  of  high  ground  that  you  would  not  otherwise  own. 
Or,  at  partner  checkpoints  if  you  didn’t  want  to  leave  forces  to  stay  the  night.  A  guy  is  “on  watch” 
within  safe  perimeter  but  shows  support  for  the  mission.  . . .  Keep  the  turret  moving.  Even  if  it 
wasn’t  fully  employed,  it’s  a  demonstration. 

Novel  employment? 

Depending  on  capabilities — armed  reconnaissance  where  you  might  not  want  to  send  people  out. 

If  you  can’t  get  a  UAV  out  due  to  weather,  depending  on  sensors,  armament,  mobility. 

Controller  needs  to  be  “buttoned  up”  somewhere.  You  need  to  be  in  the  vehicle  or  100  miles 
away. 

ESP? 

Get  an  idea  of  how  equipment  would  behave  or  be  used  in  a  certain  environment.  Driving  down  a 
road. 

Useful  to  test  the  actual  user  interface — you  are  using  the  actual  screen.  What  is  the  controller 
actually  using?  Toughbook  or  some  other  controller. 

Was  the  game  environment  real  enough  to  explore? 

Yes,  we  were  able  to  come  up  with  a  few  strategies.  They  didn’t  always  work  but  it  gave  us  a 
chance. 

What  if  you  could  manipulate  your  APD? 

Supports  the  groups  “buttoned  up”  concept  rather  than  soldier  on  the  group. 

What  kinds  of  things  were  missing  from  the  game  scenario? 

Rollover 

Noise  signature — don’t  think  it’s  realistic 
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Voice  communication  employed  during  the  session. 

In  a  real  life  scenario  if  two  controllers  where  located  across  an  active  battlespace,  they  may  have 
a  hard  time  communication  given  friction  of  radios  comms. 

If  two  controllers  were  “buttoned  up”  they  would  be  able  to  sit  next  to  each  other  and 
communicate  a  plan  in  real  time  which  is  preferred  to  a  radio  working  intermittently  in  an  urban 
environment. 

Would  you  play  this  online? 

Never  have  so  I’m  sure  if  I  would  do  that. 

If  it  was  reliable  and  didn’t  crash  it  would  be  practical. 

Would  you  play  in  a  work  environment?  In  a  sim  eenter 

If  you  get  tasked  with  a  mission  you  have  never  done  before.  If  you  could  set  the  scenario  and  pick 
your  “tools”  and  figure  out  what  you  needed  to  take  and  what  to  leave  back. 

Pre-mission. 
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APPENDIX  K.  SUMMARY  TRANSCRIPTION  OF  POST-GAME 

INTERVIEW  (SESSION  3) 


Group  3 
Defend — 

Attaeker — orientation  of  where  they  were.  The  yellow  queue  were  helpful  and  followed 
AI  eharaeters. 

Sought  out  APDs  to  fire  on,  re-load,  die,  aim,  load,  fire  for  eaeh  re-spawn 
Both  Red  and  Blue  had  AI  but  Blue  AI  avatars  help  the  effort  eonsiderably. 

**  There  was  a  guy  on  the  roof  Tried  to  take  cover;  as  soon  as  he  got  the  UGV 
controller,  the  character  stood  up  so  he  was  fully  exposed  and  easily  shot. 

Once  the  avatar  found  a  “safe”  spot,  the  APDs  were  killed  and  re-spawned  before  control 
could  be  established.  Turned  in  a  “spray  and  pray”  rather  than  clear  engagement. 

Offense  (defense)  would  have  been  mobility.  Not  a  good  idea  to  sit  right  in  front  of  the 
base.  There  wasn’t  sufficient  time  after  re-spawn  to  get  control  of  the  UGV  and  move  out 
of  the  target  area.  Ideal  employment  would  be  to  get  into  UGV,  get  it  out  of  the  target 
area,  get  it  into  defilade,  fire  and  then  establish  fields  of  fire. 

When  the  UGV/APD  was  moving,  the  enemy  was  unable  to  target  effectively  with  RPGs, 
good  self-preservation  measure. 

Attack  scenario — 

Blue — civilian  truck  to  get  a  view  of  what  was  going  on.  The  truck  was  engaged  (by  AI). 
Wanted  to  get  behind  the  prison.  There  were  a  lot  of  prison  guards  to  take  down  in  a  short 
period  of  time.  Tactic  was  to  mow  as  many  bad  guys  down  and  get  the  guards  focused  on 
the  back  side. 

Movement  in  UGV/APD  was  challenging  in  terrain.  Overestimated  the  capabilities  of  the 
vehicle;  perceived  obstacles  as  merely  shrubs  or  small  brush  rather  than  something  that 
would  inhibit  movement. 

Red — though  the  truck  was  a  red  asset  because  it  was  coming  so  quickly.  Red  figured 
out  the  way  to  “win”  was  to  kill  prisoners.  (**This  is  gaming  this  game**) 

Red  was  concerned  about  blue  taking  down  the  back  side  of  the  prison  to  allow  prisoners 
and  alternative  exit  route.  Waited  around  the  back.  Couldn’t  see  blue  team  well  until  they 
were  close. 

Wingman? 

Top  down  view  was  helpful  to  coordinate  movement.  Doesn’t  think  this  is  a  realistic  view  for 
controlling  unless  there  is  a  large  antenna  to  give  the  “god’s  eye  view”  to  allow  for  the  easy 
observation  of  movement.  Potential  to  mount  a  camera  for  visibility. 

Controller  and  soldier  and  UGV — ^player  assumed  that  he  had  multiple  assets.  Multi-tasking  was 
not  possible.  There  was  no  automation  to  allow  for  a  real  “wingman”.  You  don’t  necessarily  get 
the  full  wingman  concept.  Basically  perceived  it  as  a  packbot  or  robot  that  takes  the  player  out  of 
the  fight.  Player/so  Idler  can’t  operate  the  robot  and  move  simultaneously.  Couldn’t  run  a  scenario 
where  you  give  the  robot  direction  that  “go  here,  when  1  click  this  button,  you  are  going  to  support 
by  fire  from  this  position  while  1  go  to  the  front  gate  of  the  prison.”  If  you  could  do  that  it  would 
be  more  palpable  as  to  the  benefit  of  it.  ...  Avatar  needed  cover/concealment  because  they  were 
sucked  into  the  interface  of  driving,  guiding,  controlling,  shooting,  navigating  this  extra  entity. 
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Could  be  a  predator  operator  back  in  the  states  . . .  instead  of  getting  shot  at  could  be  in  a  nice 
comfortable  COC. 

We  didn’t  have  a  squad  leader  to  direct  action.  Four  different  guys  with  their  own  tool  as  an 
extension.  They  talked  through  a  plan  but  someone  needs  to  synthesize  the  action  because  all 
players  were  working  independently  with  their  robots.  Allows  individual  to  direct  others  where 
(who)  to  engage. 

“Map  chip”  able  to  see  where  your  peers  are  like  Blue  Force  Tracker.  Concerned  over  fratricide 
and  knowing  where  your  friends  are.  Would  like  to  see  where  you  are  going,  which  way  turret  was 
facing  in  relation  to  where  you  are  located  and  where  you  friendlies  are.  It  took  several  times  to 
get  oriented. 

A  turret  view  might  be  helpful  to  show  fields  of  fire.  You  could  extend  a  “green  line”  to  show 
range  and  fields  of  fire 

Map  chip — view  limited  to  simple  icons  (player  provided  examples). 

LINK16  SA  page — your  airplane  in  center,  move  cursor  over  the  position  of  your  fellow  aircraft 
and  it  will  give  you  status  (fuel  etc.).  If  they  engage,  you  can  see  their  fields  of  fire. 

Configuration  changes  for  robots? 

Is  there  a  unmanned  aircraft  system  (UAS)  realm  for  the  wingman  concept?  The  players  were 
familiar  with  the  Marine  Corps’  unmanned  tactical  autonomous  control  and  collaboration  system 
(UTACCS)  effort  which  incorporates  both  ground  and  air  assets. 

Quadcopter  or  some  small  vertical  takeoff  UAS  that  helps  provide  map  data.  Each  ground  system 
will  have  its  own  location  but  could  track  other  moving  targets.  Depending  of  foliage,  the  ground 
vehicle  could  lose  sight  of  target  and  they  wouldn’t  want  to  “pop  out”  right  in  the  middle  of  the 
enemy. 

Armament?  Chassis? 

Mission  dependent:  speed,  noise  is  a  factor  for  covert  missions.  Size  is  a  factor.  Fuel,  battery 
power,  power. 

Can  it  be  maintained  in  a  tactical  environment? 

If  it’s  just  a  sensor,  it  could  be  pretty  small  if  it’s  just  looking  at  stuff  but  not  approaching  bigger 
obstacles.  But  if  it’s  going  to  be  used  in  mountainous  terrain,  it  would  need  more  power. 

Armor — survive  and  RPG. 

Looking  at  prison  scenario — if  you  could  flood  prison  with  4-6  robots  and  you  could  bust  down 
gates  and  fences  each  armed  with  a  mini  gun  the  increase  capability  would  make  it  difficult  for 
even  90  bad  guys  clustered  all  around.  APD  was  very  vulnerable  to  RPG  strike — game  over.  How 
would  this  robot  defend  itself?  What  are  passive  ways  to  protect  it? 

Family  of  systems — ^reconnaissance  asset  the  size  of  a  remote  control  car  size,  quietly  into  the 
bush  giving  a  live  feed.  The  bigger  vehicle  might  need  more  SA  on  what  it’s  doing. 

APD  Chase — family  of  systems — how  would  you  employ? 

Chase — one  asset  is  unarmed  for  reconnaissance  and  it  could  move  fast  to  be  able  to  get  to  the 
area.  We  were  trying  to  figure  out  the  controls  and  they  were  down  the  road.  Armed  variants 
would  still  be  needed.  Fast  vehicle  would  be  trail  vehicle.  Armed  variants  would  be  out  in  front  so 
they  could  engage  enemy  from  the  front. 

If  the  mission  was  limited  to  “kill  this  one  bad  guy”  a  mini-gun  might  not  be  the  best  solution. 

One  that  could  call  in  guided  bomb  units  GBU  (guided  ordnance).  One  that  is  its  own  claymore — 
mobile  lED. 
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Can  the  person  get  into  the  unmanned  ground  vehicle  (UGV)  to  move  from  one  place  to  another? 
(This  is  not  the  first  time  this  has  been  broached  by  a  player.)  Person  could  ride  with  UGV,  get 
dropped  off,  push  UGV  off  somewhere  else  to  drive  in  later  to  bring  firepower.  Other  players 
don’t  like  the  idea  “if  there’s  four  bushmaster  cannons  firing,  I  don’t  want  to  be  in  a  hut  in  some 
shantytown.” 

Ambush  seems  like  easiest  situation  to  employ — fields  of  fire  of  long  range  high-caliber  weapon 
requires  de-confiiction  of  fires. 

De-confliction  of  fires  wasn’t  an  issue  but  they  communicated. 

Slopes  were  hard  to  see — ^the  “wiggle”  maneuver  was  used  to  get  up  the  hill. 

What  would  you  do  differently  with  the  robot  that  you  wouldn’t  do  with  a  person? 

Claymore. 

Without  being  super  specific,  the  willingness  to  put  that  vehicle  at  risk  is  higher.  ...  The  threshold 
is  much  lower. 

In  prisoner  scenario,  put  the  robot  between  a  few  rocks  to  get  concealment  and  cover.  Mine  was 
not  really  covered  and  concealed — it  didn’t  take  too  long  for  the  enemy  to  train  the  RPG.  If  I  was 
a  lone  Marine  I  would  not  have  chosen  that  terrain. 

Do  you  see  opportunities  for  using  wingman  in  any  missions? 

Operations  we  have  run  over  the  past  10  years.  Why  would  I  run  a  logistics  convoy  with  40  people 
to  run  a  bunch  of  water  bottles? 

Why  not  run  three  of  these  things  together  with  the  water  bottles,  controlled  by  a  guy  at  the  main 
base  of  operations?  Reduce  the  risk  to  the  40  Marines. 

Different  ways  to  extend  ISR.  It  gets  fuzzy  when  you  start  to  combine  ISR  with  offensive 
capabilities.  Challenging  to  weaponize  the  assets.  See  ability  to  maximize  potential  for  ISR. 

Using  game  environment  to  discuss  wingman?  Do  you  think  the  game  environment  is  a 
suitabie  piace  to  expiore  ideas? 

Yes,  it’s  repeatable.  It’s  adjustable.  It’s  relatively  quick  to  change,  modify.  It’s  less  expensive  that 
building  these  things  and  taking  them  out  to  a  range  with  real  bullets,  people,  etc. 

Incorporating  a  red  force  makes  it  much  more  realistic.  The  live  red  player  is  critical  vice  having 
AI  run  the  red  position. 

You  could  get  this  out  to  the  troops  to  start  developing  TTPs  so  they  get  familiar  with  it. 

Do  you  see  this  available  via  web  as  a  viable  option? 

Yes,  if  the  asset  is  actually  coming. 

Challenge  might  be  seen  if  the  actual  asset  is  different  in  the  game  asset. 

Challenge  to  incentivize  people  to  play  in  their  own  time  based  on  competition  with  entertainment 
game. 

People  play  games  on  their  phones  and  they  buy  points  to  get  capabilities  to  “win”  a  game. 

Want  to  build  ownership  where  people  will  go  off  to  code  on  their  own  to  get  better. 

What  if  time  was  set  aside  in  a  sim  center/computer  lab  on  base? 

More  practical  solution  than  expecting  people  to  play  in  their  own  time. 

If  the  players  get  the  message  that  they  may  be  able  to  impact  how  something  is  developed. 

If  the  product  is  different  than  what  they  were  expecting,  expectation  management  would  need  to 
be  conducted. 

Modification  of  assets — being  able  to  mod  by  changing  engine  and  other  features. 
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For  demographic — needs  to  be  seamless  because  the  interest  level  wouldn’t  be  sufficient  to  get  the 
player  to  learn  how  to  employ  something  way  different  on  a  keyboard. 

What  is  PMs  were  able  to  get  this  out  there  for  people  to  provide  input? 

Flash  to  bang — will  this  impact  me  in  my  career  or  will  it  come  out  in  the  far  future? 

Uncertain  about  the  length  of  the  acquisition  process 

Example — have  unit  play,  give  feedback,  return  a  week  later  and  new  capability  is  present. 
Concern  expressed  over  the  change  provided  not  matching  what  they  were  really  looking  for. 

Users  may  not  be  satisfied  because  what  was  provided  didn’t  really  match  what  they  were  looking 
for. 

If  there  are  1 8  suggestions,  pick  the  top  2  (or  whatever)  and  check  the  changes  out  a  month  later 
and  refine  what’s  provided.  Constant  development  refinement  rather  than  you  are  stuck  with  what 
you  have.  Developmental  input  would  matter  along  the  way  but  for  fielding,  “packaging  is 
importanf’ 

Here  was  our  original  solution — we  had  1000  people  play  and  provide  input  and  we  got  the  top  X 
inputs.  We  couldn’t  take  all  ideas,  “my  idea  didn’t  fare  so  well”  but  the  other  ideas  were  addressed 
and  maybe  my  idea  no  longer  makes  sense  or  maybe  it  will  be  incorporated  in  the  future. 
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APPENDIX  L.  SUMMARY  TRANSCRIPTION  OF  POST-GAME 

INTERVIEW  (SESSION  4) 


Group  4 

Red  wasn’t  sure  about  the  blue  Toyota 

Prison  attaek — like  it  best  beeause  it  was  the  most  realistie  (and  blue  won!). 

Defend — re-spawn  was  not  ideal. 

Blue  plan — get  high  ground  to  get  observation  on  the  prison.  Left  APDs  and  took  trueks. 
The  APD  blew  up.  Players  were  able  to  stay  alive  long  enough  in  the  hills. 

Red  plan — went  outside  of  the  prison  beeause  they  knew  they  were  eoming  from  the 
north.  For  the  seeond  attempt  they  stayed  eloser  to  the  prison. 

They  thought  it  was  more  realistie  to  not  know  exaetly  where  you  were  loeated.  Don’t 
think  that  some  of  the  games  for  entertainment  are  based  in  realism — they  give 
instantaneous  feedbaek  on  loeation. 

Felt  disoriented;  tried  to  mateh  terrain  to  the  map  rather  than  have  loeation  information 
handed  to  you. 

For  red,  no  indieator  that  RPG  was  aetually  loaded  by  graphies. 

Indieator  for  turret  would  be  helpful.  Like  in  an  LAV,  2  rings. 

Immediately  turn  off  safety  in  game — this  might  teaeh  bad  habits  for  real  life.  Musele 
memory  in  real  life  allows  for  rapid  transition.  If  there  is  no  value  to  the  game,  disregard 
having  safety  on. 

What  do  you  think  of  wingman?  Do  you  think  it’s  useful  to  have  a  robot? 

Depends  on  scenario  and  what  kind  of  robot. 

Need  3  people.  Need  someone  to  control  the  robot  and  two  others  to  provide  overwatch  for  the 
robot  because  the  controller  gets  sucked  in  on  the  robot. 

One  problem  in  the  game,  went  on  to  the  roof  and  got  into  the  prone.  But  once  APD  controller 
used,  avatar  stood  up  revealing  player  and  then  he  got  killed. 

Use  a  robot  to  send  across  a  danger  area  first.  Or,  provide  overwatch  from  an  area  where  you  don’t 
want  to  send  Marines. 

Scenarios  for  robots? 

Defense. 

Attack — there  are  too  many  audibles. 

We  wouldn’t  drop  1-2  guys  with  a  robot — ^not  realistic. 

If  the  robot  dies  it’s  not  a  Marine. 

Keep  the  human  somewhere  else  within  the  control  range  of  the  robot. 

If  course  of  action  (COA)  is  to  drop  LCpl  Banatz  behind  EN  lines  with  a  robot  -  NOT  realistic. 
Useful  for  urban  terrain  for  MOUT.  Awesome  to  be  able  to  put  robots  places  you  don’t  want  to 
send  Marines. 

But  how  much  gear  do  you  want  to  hump? 

Like  in  Mojave  Viper,  you  get  packbots  and  need  to  incorporate  them  into  the  scheme. 

Humping  another  weapon,  ammo,  batteries.  . . .  Just  throw  in  the  back  of  a  truck. 

But  if  you’re  working  with  tanks  to  support  infantry.  But  if  there  aren’t  enough  tanks,  you  could 
employ  this  as  an  alternative  with  slightly  different  objective — down  small  allies  or  into  buildings. 
How  far  forward  can  you  push  a  robot? 
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Can  the  robot  do  something  that  you  wouldn’t  get  by  support  by  fire — we  already  have  precision 
support  by  fire — GPS  artillery  rounds.  Counter  was  that  this  is  actually  a  direct  fire  weapon. 

If  player  is  moving,  he  can’t  control  the  robot  so  it’s  not  really  a  wingman,  it’s  an  extension. 
Imagine  you  are  clearing  a  sector  of  a  city,  you  could  use  it  like  tank  and  infantry  integration.  But, 
to  counter,  the  logistical  requirements  to  support  the  tanks  (fuel,  water,  mechs).  But  what  other 
gear  do  you  need?  Fuel,  batteries. 

Brings  firepower,  armor. 

Can  we  blow  in  place  if  it  is  dead  in  the  water? 

Defense. 

Gun  truck —  .50  cal  in  the  fight  where  you  might  not  have  had  one  previously.  Rather  than  a  high 
mobility  multi-purpose  wheeled  vehicle  (HMMWV)  with  a  .50cal  you  have  a  robot.  T72  is 
automated  reload  -  concern  about  weapon  jams — gun-truck  is  out  of  the  fight. 

Armament? 

Hesitant  to  put  a  MK19.  Jamming,  ROE  considerations — only  appropriate  for  full  spectrum.  Or, 
like  a  track —  .50cal  and  MK19  together. 

Fields  of  fire  a  concern  for  chase  but  not  for  the  other  scenarios. 

Would  need  to  develop  an  institutional  change — it’s  a  robot,  if  it  gets  shot,  it  gets  shot. 

A  UAV  currently  gets  treated  “like  a  person”  because  it’s  an  important  asset.  Recovery  of  UAV  is 
prioritized. 

Tactical  control  measures  applied.  Gravity  of  situation  determines  the  amount  of  risk  you  want  to 
take.  But  if  it’s  a  gnarly  situation  it  will  be  easier  to  put  a  robot  into  the  fight. 

What  is  the  interaction:  laptop  or  full  heads  up  display  full  immersion? 
o  Two  people — one  person  driving  and  one  manning  turret. 

o  Or,  have  controllers  back  in  “vegas”  with  more  specific  controls.  Having  a  heads  up 
display  where  you  turned  your  head  would  make  it  a  useful  technology.  Security  would 
not  be  an  issue. 

o  Or,  push  controls  as  far  forward  as  possible.  Local  security  is  important.  Questions  ability 
to  integrate  a  heavy  weapon  into  the  scheme  of  maneuver  without  being  able  to 
physically  reach  out  and  tough  the  controller, 
o  Counter — no  different  than  controlling  support  by  fire  over  radio, 

o  Must  take  into  account  the  additional  security  requirement  for  additional  people  in  the 
headquarters  (HQ)  or  a  fire  support  team  (FST). 

Trial  and  error — where  loeate,  how  to  employ  assets?  ESP  to  develop  ideas? 

Good  because  you  don’t  have  the  overhead  of  going  to  the  field  and  lock  on  all  logistical  support 
requirements. 

Need  to  make  it  as  realistic  as  possible. 

Integrate  into  a  mission  for  a  squad  or  platoon. 

Can  APD  be  manually  controlled  if  all  other  controls  fail? 
o  Could  the  person  control  it? 

Customization  option? 

VBS2  -  can  it  be  modified  by  players? 

Have  squad  leader  or  platoon  leader  to  “fight”  the  scenario.  Have  a  Marine  or  two  be  the  OPFOR 
Who  is  saying  these  need  to  be  armed?  I’m  terrified  of  robots  with  guns. 

If  you  take  the  weaponry  on,  it  will  be  smaller,  lighter,  and  require  less  logistics  load. 

Why  arm  it  in  the  first  place?  I  would  be  concerned  about  a  Marine  getting  run  over. 
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Employ  like  a  mobile  strong  point — a  Marine  is  then  gunning  it.  Not  relying  on  the  system 
sensors.  Something  that  drives  up  and  deploys  into  a  twin  gun  section.  Why  employ  it  like  a  tank 
at  all?  Small  wheeled  or  tracked  asset  (like  the  mules  that  carry  packs).  The  position  could  aid 
with  bounding  overwatch — ^this  is  like  a  transformer.  It  carries  your  weapons  and  armor 
Breaching,  smash  through  walls. 

Terrified  of  rules  of  engagement  (ROE)  with  robots  on  the  battlefield. 

What  conclusions  do  you  think  we  cannot  draw? 

Size  was  a  challenge  from  lED  robot  size  to  HMMWV. 

Personal  security  requirements  of  operator  and  how  to  integrate  the  operator  into  the  unit. 

Logistics  requirements. 

Do  you  think  ESP  is  practical  for  the  future? 

Need  to  establish  incentive  to  play. 

Each  Marine  Expeditionary  Force  (MEF)  could  send  an  independent  augment  for  a  month — they 
get  good  with  the  controls  and  cycle  through  to  test  and  evaluation.  Becomes  a  MEF  think  tank  for 
the  MEF.  Short  timers — guys  with  EASs  coming  up  that  you  don’t  want  to  incorporate 
mainstream  in  the  unit. 
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APPENDIX  M.  SUMMARY  TRANSCRIPTION  OF  POST-GAME 

INTERVIEW  (SESSION  5) 


Group  5 

Wingman — separation  between  reality  and  the  game. 

See  the  advantage  of  having  the  robot  at  the  prison. 

One  vehiele  was  in  support  by  fire  position,  Red  team  ran  out  of  the  prison  to  be  slightly 
more  offensive.  Doubted  his  ability  to  hit  the  APD  with  an  RPG — the  APD  was  headed 
to  the  game  taking  people  down. 

Red — eouldn’t  use  the  RPGs  in  the  guard  tower  beeause  the  tower  was  enclosed  and 
there  were  people  close  by.  Thought  he  was  observed  in  a  guard  tower  but  had  several 
opportunities  to  shoot.  Red  realized  that  the  APD  couldn’t  hear  or  see  anything. 

**  While  robot  is  out  in  an  environment,  it  can’t  “hear.” 

Near  miss  indicator?  May  or  may  not  be  realistic;  camera  on  the  periphery  to  be  able  to 
see  left  and  right  if  there  was  a  round  impacting  nearby.  Since  Red  had  the  opportunity  to 
shoot  several  times  without  the  APD  moving,  it  was  obvious  to  the  Red  player  that  the 
APD  operator  didn’t  know  that  the  robot  was  being  engaged. 

Robot  is  an  extension  of  the  player  but  it  doesn’t  have  the  same  senses  as  a  person. 

Other  scenarios? 

From  a  real  life  perspective.  A  lot  can  go  wrong  with  the  robot  out  there. 

IT  was  pretty  easy  to  move  the  robot  around  on  the  terrain,  perception  that  it  wouldn’t  be  that  easy 
in  the  real  world. 

You  are  more  removed  from  the  scenario.  People  in  video  games  can  be  pretty  ruthless  because 
they  don’t  care.  Players  are  more  aggressive,  less  risk  averse.  Moral  and  ethical  concerns. 

Games  for  entertainment  vs.  games  for  training — need  to  input  constraints  to  encourage  risk  but 
not  manipulate  players’  inputs  beyond  reasonable  courses  of  action  (COA). 

Player  rated  his  own  performance  as  over-aggressive. 

User  of  robots:  if  integrated  in  with  other  forces,  rather  than  pure  robots  or  pure  human  forces. 

Two  robots  are  out  there  on  their  own  and  one  is  down — need  to  do  a  TRAP  to  recover  it  or  self- 
destruct  it. 

Advantage  seen  as  integrated  into  unit.  Example:  company  size  element  to  secure  a  prison. 
Emplacing  robots  in  high  ground  in  support  by  fire  positions.  Having  personnel  come  in  behind 
the  assets  once  threat  level  reduce  to  a  more  acceptable  level.  This  would  reduce  threat  to  a  more 
acceptable  level.  The  controllers  would  be  co-located  with  the  company  fire  support  coordination 
center  or  with  a  Fire  Support  Team.  The  commander  would  then  be  able  to  direct  vehicles  to 
particular  positions  to  have  fires  directed  at  a  particular  target  location. 

Wingman — a  section  is  two  vehicles.  One  vehicle  be  human  operated  and  one  be  the  UGV 
controlled  by  someone  sitting  in  the  manned  vehicle.  This  is  basically  “doubling  assets”  without 
doubling  number  of  people  endangered  in  the  scenario. 

How  would  you  change  the  firepower,  armament,  mobility? 

Concerns  over  remotely  controlling  a  vehicle  in  a  combat  zone. 

Tracked  vehicles. 

Running  into  trees  and  rocks — what  kind  of  damage  is  occurring?  Can  damage  be  built  into  the 
models  to  account  for  how  someone  is  driving  and  how  terrain  actually  impacts  the  vehicle?  What 
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is  a  normal  level  of  damage  or  wear  and  tear  for  different  terrain?  What  about  an  unskilled 
operator — how  much  worse  is  the  damage? 

Armor — group  expressed  eoncem  over  vulnerability  to  RPG 

Mix  of  armament  vs.  mobility  will  be  situational  dependent.  If  supporting  tanks  against  other 
tanks,  would  need  heavier  armor.  If  running  light  scout,  need  more  sensor,  lighter  armor,  higher 
mobility.  Without  troop  quarters,  additional  armor  can  be  incorporated  without  added  to  weight  of 
the  system.  Need  to  address  weight  and  opportunities  to  self-recover — like  throwing  track  or 
losing  wheel. 

What  do  you  prefer  lots  of  light-armored  robots  or  fewer  survivable  assets? 

Light  armored  guys  want  more  light  assets  (really  wants  medium). 

Seen  as  more  a  tool  for  reconnaissance,  to  gather  imagery.  Tend  toward  lighter. 

Some  players  would  rather  heavy  because  they  don’t  trust  the  imagery  for  ISR.  And,  thinking  of 
how  soldiers  would  treat  equipment.  Asset  needs  to  be  robust  to  abuse  and  use.  If  light  asset  could 
be  robust  to  use  and  user  abuse  then  it  might  be  an  option. 

“Loaded  question.” 

How  would  you  employ  a  robot  considering  risks? 

If  there  is  an  extremely  dangerous  scenario  where  you  can  prevent  putting  a  human  being  in  that 
harm  and  use  the  robot  to  clear  the  way  before  they  enter.  Assault  breaching  vehicle  comparison  to 
proof  a  lane  and  have  unit/forces  that  fall  in  behind  the  cleared  route. 

Concerns? 

Situational  awareness.  Like  the  idea  of  sound. 

Marine  Corps  is  more  likely  to  support  infantry — having  ground  forces  able  to  shout  and  have  the 
operator  hear  the  interaction  and  make  adjustments. 

The  operator  is  in  theatre,  in  the  area  of  operations  (AO).  Not  in  Nevada  because  they  lose 
connectivity  to  the  fighting  environment.  Prefer  with  the  commander  on  the  ground  who  is 
directing  forces. 

ESP — is  a  game  environment  a  suitable  place  to  be  experimenting  with  ideas  like  this? 

Cost  effective  way  to  do  it. 

Controlled  live  environment  to  test  it  in  concert  with  the  game  environment. 

Concerns  about  the  physics  of  the  environment. 

This  is  a  good  forum  for  brainstorming  new  ideas ! 

What  do  you  wish  you  had?  I  wanted  a  widget  like  this  . . .  and  then,  be  able  to  try  it  out  and  see  if 
it  is  practical  or  not. 

This  is  preferable  than  taking  all  the  “good  idea  fairy”  ideas  and  making  physical  prototypes. 
Testing  environment  to  determine  what  is  viable  for  physical  testing. 

What  do  you  think  you  can’t  draw  from  the  game  environment? 

Situational  awareness. 

Physics. 

Damage. 

Natural  friction  that  goes  along  with  actually  going  out  and  performing  a  mission, 
o  Crypto  goes  down, 
o  Controller  isn’t  interfacing  with  vehicle, 
o  Deadline  pressures  of  mission  requirements. 
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o  Being  able  to  proceed  with/without. 

o  Perspective  of  enemy — how  it  impacts  their  OODA  loop.  Do  they  adjust  their  tactics? 
Red  would  be  intimidated  that  robots  were  coming  at  them  rather  than  real  live  people. 

■  Possible  mitigation  is  to  bring  live  red  players. 

Do  you  think  you,  your  Marines,  can  have  an  impact  on  future  development? 

If  the  people  that  make  decisions  are  open  minded  and  buy-in  to  a  technique  like  this. 

Analysis  of  Alternative — scope  down  to  comparison 

Option  for  PM — there  is  merit. 

This  is  a  different  method  of  conducting  simulation:  constraints,  limitations,  assumptions. 
Command  needs  buy-in  for  how  evaluation  is  conducted. 

Buy-in  a  lower  levels — taking  soldiers/Marines  to  sim  centers 

o  Part  of  broader  study. 

o  Competing  demands — ^programming  time  to  assist  with  an  effort, 
o  Marine  Corps  Warfighting  Lab  (MCWL) — experimental  division,  think  tank,  having 
better  representation  in  an  organization  like  that.  Or,  another  option  is  augmenting  with 
units  when  they  are  on  a  down-side  of  training  schedule. 

Home  play? 

1  don’t  think  players  would  play  at  home  without  significant  incentive  due  to  challenge  to  compete 
with  commercial  gaming  options. 

If  you  can  download  VBS2  light  at  home — not  sure  how  to  collect  metrics. 

Think  that  unit  participation  is  more  likely. 

What  are  you  doing  to  incentivize  individuals  to  play?  Points  on  AKO  or  USMC  site  to  reward 
their  participation. 

Part  of  training  package — schedule  rather  than  an  add-on  for  a  unit  to  jump  on. 

Incentive  piece — more  realistic  usually  means  less  fun.  How  to  grow  individual  motivation  to 
participate? 

If  soldiers  were  to  play  and  see  a  return — the  turn  around  is  the  challenge.  A  private  might  see 
their  effort  when  they  reach  the  rank  of  general — ^the  immediate  feedback. 

IDEA  from  previous  group — top  20  recommendations  published  so  that  soldiers  could  see  their 
ideas. 

Maybe  warfighting  lab  brings  people  to  Quantico  to  help  with  their  ideas  if  they  are  good 
players — this  is  a  challenge  with  anonymity.  Maybe  they  can  opt  out  if  they  don’t  want  to  partake. 
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